How the Ledger measures
The Standard asks whether an agent is correct, and settles it by assertion. The Ledger asks whether an agent is better than doing the work yourself, which cannot be settled by assertion — so it is settled by a rubric written before anyone has seen an answer.
The shape of a benchmark
A benchmark is one task, put to two arms: a Marque agent, and a human analyst working by hand. Both answer the same task, which is meant to be pinned to the same block. Each arm is run twice, because a single run is an anecdote.
The agent arm records the block it read. The human arm, run by hand, recorded none — but the block is not lost: each frozen benchmark task contains exactly one BSC block number, and every human run’s manifest hash matches that frozen task. So the block is recovered from the immutable task and recorded in an append-onlyprovenance supplement. The raw run row is never rewritten; the recovered value is shown beside the original (not recorded → effective 120…) with the source task hash, so a reader can check the derivation.
Where the agent arm needs re-running against a block older than public nodes retain, it is replayed against that exact block through an archive RPC endpoint, and the run is rejected unless the engine reports it read the pinned block. A replay is not a reproduction: it answers the same frozen task at the same immutable state as the human arm, and only the agent side was executed later. The card says so plainly.
Before either arm runs, four things are frozen and hashed: the task, the inputs, the rubric, and the rubric’s version. Those hashes are published on the benchmark page. A rubric that can be edited after the answers arrive is not a rubric, so the registration timestamp is recorded and never touched by a re-run.
The human arm is recorded
The manual arm is a person doing the task by hand, on camera. Each benchmark links a screen recording of the analyst reading the contract on BscScan, querying pool state with cast call, or working the health factor — pinned to the benchmark block. The recordings are shown on each benchmark page, not hidden behind a link, because “a human really did this” is a claim that deserves to be seen rather than asserted. The result the analyst produced is pasted into the intake, which hashes it into the manifest exactly as the agent’s answer is hashed.
Blind grading
Both outputs have their source labels stripped before scoring. The grader sees two answers and the rubric, and does not know which arm produced which. The runner is deliberately incapable of scoring: an arm graded at the moment it was produced would be graded with the label in plain sight, which is the failure mode the whole design exists to avoid.
Timing
Wall clock, always, never an estimate. The agent arm is timed around the same public HTTPS request a buyer would make — including TLS, the reverse proxy and the SSRF guard, because a buyer waits for those too, and excluding them would flatter the agent. The manual arm is timed by the analyst’s own stopwatch, and the analyst states in their own words how they measured it. That statement is published with the result.
Cost
Itemized, never a single figure: gas, model spend, agent fee, and human time as separate lines, with the hourly rate recorded so a reader can re-price the result against their own. Where a line is genuinely zero it is zero, and where it is unknown it is not filled in. The agent arms below spent no gas and no model budget, because the engines are deterministic arithmetic over chain reads — that is why those lines read zero rather than unknown.
Sittings
Re-running an arm does not overwrite the previous run. Each sitting is recorded separately, named for the block it started at, and the Ledger publishes the latest one. Earlier sittings are kept and counted, because a run is a measurement we made and cannot recreate. Two sittings read different chain state, so their repetitions are not interchangeable and are never pooled or chosen between.
The reproduce button on a benchmark page runs the agent arm live against today’s block. It is recorded as a reproduction and is deliberately excluded from the published result — otherwise anyone could replace a registered comparison by pressing a button.
Sealed calls
Separately from the benchmarks, every recommendation a Marque agent issues is hashed and written to MarqueRegistry.sealCall at the moment it is issued. The rule that will decide the call is written into the same payload, before the outcome is known, so it cannot be softened later to make a call look right. A scorer resolves each call against subsequent chain state once its window has elapsed.
An agent profile shows the count, the window, how many have resolved, and the breakdown of outcomes. It never shows a bare win rate. With a handful of calls a percentage is a number that looks like evidence and is not, so a small sample is labelled as one rather than rounded into a headline.
What makes a result void
- A rubric edited after either arm ran. The hash would change and the mismatch is public.
- An arm whose block cannot be read by the other arm, so the two answered different states.
- A manual arm without a stated timing method, or without its elapsed time.
- Any arm we simulated. The manual arm is run by a human or it does not exist.
What is not settled here
The Ledger measures judgement, speed and cost. It does not measure correctness — that is the Standard, and a check belongs there only if it can be written as an assertion with a numeric tolerance. Nothing that requires an opinion is ever graded as conformance, and nothing with one right answer is ever graded here.