How the Ledger measures
The Standard asks whether an agent is correct, and settles it by assertion. The Ledger asks whether an agent is better than doing the work yourself, which cannot be settled by assertion — so it is settled by a rubric written before anyone has seen an answer.
The shape of a benchmark
A benchmark is one task, put to two arms: a Marque agent, and a human analyst working by hand. Both answer the same task, pinned to the same block. Each arm is run twice, because a single run is an anecdote.
Before either arm runs, four things are frozen and hashed: the task, the inputs, the rubric, and the rubric’s version. Those hashes are published on the benchmark page. A rubric that can be edited after the answers arrive is not a rubric, so the registration timestamp is recorded and never touched by a re-run.
Blind grading
Both outputs have their source labels stripped before scoring. The grader sees two answers and the rubric, and does not know which arm produced which. The runner is deliberately incapable of scoring: an arm graded at the moment it was produced would be graded with the label in plain sight, which is the failure mode the whole design exists to avoid.
Timing
Wall clock, always, never an estimate. The agent arm is timed around the same public HTTPS request a buyer would make — including TLS, the reverse proxy and the SSRF guard, because a buyer waits for those too, and excluding them would flatter the agent. The manual arm is timed by the analyst’s own stopwatch, and the analyst states in their own words how they measured it. That statement is published with the result.
Cost
Itemized, never a single figure: gas, model spend, agent fee, and human time as separate lines, with the hourly rate recorded so a reader can re-price the result against their own. Where a line is genuinely zero it is zero, and where it is unknown it is not filled in. The agent arms below spent no gas and no model budget, because the engines are deterministic arithmetic over chain reads — that is why those lines read zero rather than unknown.
Sittings
Re-running an arm does not overwrite the previous run. Each sitting is recorded separately, named for the block it started at, and the Ledger publishes the latest one. Earlier sittings are kept and counted, because a run is a measurement we made and cannot recreate. Two sittings read different chain state, so their repetitions are not interchangeable and are never pooled or chosen between.
The reproduce button on a benchmark page runs the agent arm live against today’s block. It is recorded as a reproduction and is deliberately excluded from the published result — otherwise anyone could replace a registered comparison by pressing a button.
Sealed calls
Separately from the benchmarks, every recommendation a Marque agent issues is hashed and written to MarqueRegistry.sealCall at the moment it is issued. The rule that will decide the call is written into the same payload, before the outcome is known, so it cannot be softened later to make a call look right. A scorer resolves each call against subsequent chain state once its window has elapsed.
An agent profile shows the count, the window, how many have resolved, and the breakdown of outcomes. It never shows a bare win rate. With a handful of calls a percentage is a number that looks like evidence and is not, so a small sample is labelled as one rather than rounded into a headline.
What makes a result void
- A rubric edited after either arm ran. The hash would change and the mismatch is public.
- An arm whose block cannot be read by the other arm, so the two answered different states.
- A manual arm without a stated timing method, or without its elapsed time.
- Any arm we simulated. The manual arm is run by a human or it does not exist.
What is not settled here
The Ledger measures judgement, speed and cost. It does not measure correctness — that is the Standard, and a check belongs there only if it can be written as an assertion with a numeric tolerance. Nothing that requires an opinion is ever graded as conformance, and nothing with one right answer is ever graded here.