Evaluating a Mina staking pool’s performance

Mina uses a proof of stake consensus mechanism with a block producer selection similar to Ouroboros, known as Ouroboros Samisika. The opportunity to produce a block for a slot is determined by a verifiable random function (VRF). This selection can be thought of like a lottery with each block producer independently running this function for each slot. If they get a VRF output greater than a threshold proportional to the producer’s stake, they get the chance to produce a block at the designated slot.

The VRF function takes as input a random seed in addition to the producer’s private key and current stake distribution and is deterministic. The random seed is obtained by hashing the output of the VRF function of block producers for the first 2/3rd of the previous epoch. So the output from the blockchain itself becomes the randomness for the next epoch (which is the origin of the name Ouroboros).

The slots won are only known to the block producer, which aids security as it is impossible for an adversary to target a known block producer at a certain slot, e.g., by a denial of service or targeted attack.

One issue with the process is that it is challenging to infer a staking pool’s performance reliably. Current approaches focus on “luck” by comparing the expected performance of the delegator with that observed. While this works well when averaged over long periods, it is unreliable over shorter durations and with lower stakes where luck can play a significant role. A more robust approach to determining the performance of a delegate would be to compare the number of slots won to the number of blocks produced at the canonical height of the chain.

In the latest version of the Mina client (1.2.0+), new tooling was added that allows a block producer to generate witnesses to attest to the slots they won for an epoch. These witnesses can be independently verified and may be used to evaluate a delegate's performance reliably.

A block producer may also use this tooling to determine all slots they won in advance of an epoch.

Additional VRF commands

There are new commands in the daemon under mina advanced vrf for generating and verifying VRF witnesses, either individually or via a batch:

• generate-witness
• batch-check-witness
• batch-generate-witness

In addition, there are two new GraphQL endpoints. These are useful for generating/checking a small number of witnesses and won’t be applicable due to performance reasons of evaluating many slots and delegators, which is the article’s primary focus.

• checkVrf
• evaluateVrf

Determining all slots won in an epoch

The most common use-case for the new commands is for block producers to determine in advance all the slots they won for the current and next epoch (when finalized).

As inputs to the batch-generate-witness and batch-check-witness commands we will need:

• Global slots to evaluate.
• Epoch seed for the epoch.
• The total stake of the epoch staking ledger.
• Account indexes for all delegators.
• The stake of each delegator, in the staking ledger.

While it is possible to obtain all of these inputs manually by extracting the required data from a node, there is an excellent tool built by @kobigurk of the ZKValidator pool, written in Rust, that will gather and format these inputs. To use the tool, you will need:

• Rust installed.
• A copy of the repo.
• A node running at least 1.2.0 (beta versions work) with GraphQL accessible.

As an example, we’ll determine all slots won for the MinaExplorer pool for epoch 0.

1. Install Rust

The simplest method to install Rust is to use rustup from here.

2. Clone the repo

The GitHub repo is located here, so clone into the current directory via:

git clone https://github.com/zkvalidator/mina-vrf-rs.git

3. Generate the requests

The first step of running the tool is to generate the inputs to produce the witnesses. This is comprised of all the slots and delegators we wish to evaluate for a given epoch. To generate the inputs for epoch 0 for the key B62qpge4uMq4Vv5Rvc8Gw9qSquUYd6xoW1pz7HQkMSHm6h1o7pvLPAN and save to a filename requests run:

cargo run --release -- batch-generate-witness --pub B62qpge4uMq4Vv5Rvc8Gw9qSquUYd6xoW1pz7HQkMSHm6h1o7pvLPAN --epoch 0 > requests

This step generates the inputs required for the next stage, as shown below. If you are doing this manually, you will need to create these JSON objects and determine the delegatorIndex for each delegation via GraphQL.

{
   "globalSlot":"7140",
   "epochSeed":"2vaMKZSXxYDaUzYVRZBNVRmwW7JwXE3vS3NKZfR8d1aijpZU2t3q",
   "delegatorIndex":5
}
{
   "globalSlot":"7140",
   "epochSeed":"2vaMKZSXxYDaUzYVRZBNVRmwW7JwXE3vS3NKZfR8d1aijpZU2t3q",
   "delegatorIndex":6
}

4. Generate the witnesses

Next, we pass these requests into the mina advanced batch-generate-witness command to generate the witnesses. This step requires access to the private key of the block producer, and it can be quite slow for a large number of delegators.

cat requests | mina advanced vrf batch-generate-witness --privkey-path /keys/my-wallet | grep -v CODA_PRIVKEY_PASS > witnesses

An example generated witness is shown below, with one created for every slot for every delegator.

{
  "message": {
    "globalSlot": "21420",
    "epochSeed": "2vafLYumHvDQP49XQScJnPFqh977FdAsjhSrZbrtrXpiPsydG5Sj",
    "delegatorIndex": 5
  },
  "publicKey": "B62qpge4uMq4Vv5Rvc8Gw9qSquUYd6xoW1pz7HQkMSHm6h1o7pvLPAN",
  "c":
    "27756566863414797581781478807131942238852341092608514702602203744130892819006",
  "s":
    "19521050341837802368631492777160646043301977200100017703727282250002882104946",
  "ScaledMessageHash": [
    "0x886ad33bd0866191c79a5439f511a6cb4cacef4188805420af9782348a49fd1f",
    "0xdd003e9ad3d106fa3ae8337d3c638f87131b2ed4592bc7b971637675af5f5017"
  ]
}

5. Patch the witnesses with delegation totals

The next step involves adding in the balances from the staking ledgers. The account balances and the total stake of the epoch are used to determine if the VRF threshold has been met, i.e., the delegator has won the right to produce a block for that slot.

cat witnesses | cargo run --release -- batch-patch-witness --pub B62qpge4uMq4Vv5Rvc8Gw9qSquUYd6xoW1pz7HQkMSHm6h1o7pvLPAN --epoch 0 > patches

This creates a file with the witness data appended with the balance information from the relevant staking ledger.

{
   "vrfThreshold":{
      "delegatedStake":"67869.906422964",
      "totalStake":"829489852.840039300"
   }
}

6. Evaluate the witness data

Next, we will pass the patched witness data to the batch-check-witness command to check the witnesses and verify if the VRF threshold has been met for the specified stake.

cat patches | mina advanced vrf batch-check-witness | grep -v CODA_PRIVKEY_PASS > check

This command outputs a file named check with the result of the evaluation for each slot and delegator.

{
  "message": {
    "globalSlot": "0",
    "epochSeed": "2va9BGv9JrLTtrzZttiEMDYw1Zj6a6EHzXjmP9evHDTG3oEquURA",
    "delegatorIndex": 6
  },
  "publicKey": "B62qpge4uMq4Vv5Rvc8Gw9qSquUYd6xoW1pz7HQkMSHm6h1o7pvLPAN",
  "c":
    "5471098132471102051558687182262111978937438362416021679410512408349591978876",
  "s":
    "3617882623492222290347674815508142437068468594088059083619705873549466051502",
  "ScaledMessageHash": [
    "0x9285b788681be702808ae0ce96b76cb9794d0ae61012bc1d7c82eef63b9b5b31",
    "0x69a767e6fda7f0db147fbf5bba55d65357b423b552b8b767a3caf278ffed8222"
  ],
  "vrfThreshold": {
    "delegatedStake": "32558",
    "totalStake": "805385692.8400393"
  },
  "vrfOutput": "EiRtSPahRsDAvbev8Ua5UbqQ4s1G9AciUgJk1NYfah6oMk2iW1vBD",
  "vrfOutputFractional": 0.6448230137594151,
  "thresholdMet": false
}

The line we are most interested in "thresholdMet": false.This will be true if the slot (for this delegator) was a winning one.

7. Returning the slots won

The final step of the process outputs a summary of the prior step providing a list of all slots won during the epoch (both global slot and slot during epoch). If invalid witness data were provided, these slots would be listed in invalid slots.

cat check | cargo run --release -- batch-check-witness --pub B62qpge4uMq4Vv5Rvc8Gw9qSquUYd6xoW1pz7HQkMSHm6h1o7pvLPAN --epoch 0

For the MinaExplorer pool for epoch 0, this corresponds to the following output. As this is epoch 0, the local and global slots are the same.

invalid slots: []
invalid local slots: []
producing slots: [47, 48, 91, 196, 228, 247, 266, 300, 373, 378, 407, 478, 577, 596, 605, 648, 649, 863, 873, 914, 969, 1037, 1099, 1102, 1112, 1133, 1159, 1209, 1268, 1274, 1309, 1354, 1499, 1540, 1616, 1618, 1693, 1744, 1762, 1872, 1927, 1971, 2027, 2077, 2193, 2196, 2452, 2511, 2523, 2531, 2551, 2573, 2655, 2664, 2665, 2672, 2678, 2712, 2724, 2769, 2781, 2925, 2943, 2990, 3025, 3048, 3110, 3143, 3173, 3180, 3249, 3279, 3333, 3345, 3445, 3534, 3544, 3578, 3677, 3761, 3781, 3817, 3881, 3919, 4015, 4034, 4072, 4121, 4223, 4240, 4269, 4332, 4342, 4361, 4451, 4606, 4639, 4655, 4717, 4730, 4746, 4827, 4830, 4953, 4961, 4994, 5042, 5139, 5141, 5163, 5213, 5269, 5367, 5431, 5459, 5498, 5503, 5587, 5698, 5834, 5855, 5872, 5900, 5973, 6012, 6203, 6217, 6298, 6331, 6365, 6466, 6570, 6575, 6712, 6796, 6805, 6818, 6846, 6945, 6981, 7045, 7100]
producing local slots: [47, 48, 91, 196, 228, 247, 266, 300, 373, 378, 407, 478, 577, 596, 605, 648, 649, 863, 873, 914, 969, 1037, 1099, 1102, 1112, 1133, 1159, 1209, 1268, 1274, 1309, 1354, 1499, 1540, 1616, 1618, 1693, 1744, 1762, 1872, 1927, 1971, 2027, 2077, 2193, 2196, 2452, 2511, 2523, 2531, 2551, 2573, 2655, 2664, 2665, 2672, 2678, 2712, 2724, 2769, 2781, 2925, 2943, 2990, 3025, 3048, 3110, 3143, 3173, 3180, 3249, 3279, 3333, 3345, 3445, 3534, 3544, 3578, 3677, 3761, 3781, 3817, 3881, 3919, 4015, 4034, 4072, 4121, 4223, 4240, 4269, 4332, 4342, 4361, 4451, 4606, 4639, 4655, 4717, 4730, 4746, 4827, 4830, 4953, 4961, 4994, 5042, 5139, 5141, 5163, 5213, 5269, 5367, 5431, 5459, 5498, 5503, 5587, 5698, 5834, 5855, 5872, 5900, 5973, 6012, 6203, 6217, 6298, 6331, 6365, 6466, 6570, 6575, 6712, 6796, 6805, 6818, 6846, 6945, 6981, 7045, 7100]

Determining the performance of a delegate

Since publishing https://github.com/zkvalidator/mina-vrf-rs has added support for evaluating the blocks produced compared to the canonical chain, which is a more comprehensive evaluation than detailed below. The below is left for reference.

In the above example, only step 4 requires the private key of the delegate. This means the block producer can generate the witness data and pass it to anyone else to verify. A potential use-case for this is for a block producer to demonstrate they have produced blocks for all slots they won.

The MinaExplorer pool publishes the complete witness data for all epochs on this page (link to witness file is in the slots column for the epoch)

However, producing a block in a slot isn’t the only metric we should consider when measuring performance. If the nodes are not well maintained, they may not produce a block at the correct height, i.e., they are out of sync. Therefore we should also consider if the delegate produced a block at the current canonical height. As there can be many blocks produced for the same slot by multiple producers, we cannot simply rely on the number of canonical blocks produced during the epoch.

This script takes as input the slots won from the final step and checks to see whether the block producer produced at the canonical height for each slot. As a result, we can calculate a performance score based on the actual number of blocks won and produced. This could be developed to measure performance better rather than relying on luck-based metrics. For example, the evaluation for the MinaExplorer pool for epoch 0 returns:

There were 142 slots and the delegator produced at the canonical height for 142 slots and has a performance score of 100.00%

Note this implementation is subject to short forks/reorgs where the block producer may not have seen the latest block and can likely be improved.

The curtailed output of running the script for MinaExplorer Pool for Epoch 0