Systems and rounds
A family is one leaderboard: a fixed set of cases or tasks and the systems that answer them. A system is a model with its effort, harness, and prompt, and changing any part makes a new system. Each round is a dated snapshot that refits every result so far. Families are grouped by aspect, the skill they measure.
Votes
With sides counterbalanced, the rater picks the better output of each pair, a tie, or both bad.
Gates
A failing output loses its pairs and two identical outputs tie, so these pairs settle without a vote.
| Gate | Meaning |
|---|---|
| Empty | The output is blank. |
| Truncated | The model stopped before finishing, as at its output limit. |
| Length | The output falls outside the family's character bounds. |
| Language | Most of the prose and code comments are in a language other than the family's. |
| Script | The output uses Traditional-only characters that the prompt doesn't. |
Point scores
A penalized Bradley-Terry model fits Elo-like point scores to human votes and settled pairs. Ties and both-bad votes count half a win. Each round records its human weighting policy. Ledger rounds cap each actor's complete family contribution before fitting category or component subsets, and use repeats only for consistency diagnostics. Historical rounds may record unit weights. Each settled output pairing receives its share of that case's sample pairs.
Compare scores within one round of one family. Each closed board records its point method, anchor, weighting policy and coverage rules under Round settings. Categories refit their own comparisons while retaining that round's settings.
Reading the leaderboard
| System | Point score axis | Score |
|---|---|---|
| Measured | 1,040 | |
| Preliminary | 930, preliminary | |
| Far behind | 380 | |
| No votes yet | no votes |
- Point score
- Preliminary
- Score below the axis
The axis starts at most 600 points below the top score.
| Column | Meaning |
|---|---|
| Rank | The order of fitted point scores within this round. Equal scores share a rank. |
| Anchor | The system every score is relative to: the family's reference, or the most-voted system when a round cannot rank it. |
| Preliminary | A coverage marker for systems below the round's recorded decisive-vote threshold. |
| Unranked | Listed below the table, unscored, until comparisons connect it to the anchor. |
| Votes | The rater's votes on the system, leaving out pairs settled without a vote. |
| Both bad | The share of its votes judged both bad. |
| Gate failures | The share of its outputs that failed an automatic check, such as being blank or in the wrong language, and so lost their pairs without a vote. |
| Cost / output and p50 latency | The mean cost at list prices and the median time to the complete response. |
A dash marks a missing value. A category view refits that category's comparisons, while columns labeled "all categories" retain family-wide values. Preliminary marks use the round's recorded raw decisive-vote threshold and describe coverage.
Uncertainty
Uncertainty for new cases and voters is unavailable until the method is calibrated.
Point ranks describe the fitted order. They supply no confidence interval, population separation claim or leader decision. More votes alone do not validate a population uncertainty method.
Round summary
| Figure | Meaning |
|---|---|
| Rater agreement | How often a side-swapped repeat matched the original vote. This describes repeat consistency. |
| Split-half τ | Kendall's τ between the orders fit to odd and to even sessions, where 1 means the same order. |
| Left out of the fit | Votes on a retired case, on an output since removed or now failing a gate, or on a pair the outputs now settle. |
| Length bias | How far the rater's votes lean toward the longer or the shorter output, in score points per standard deviation of the pair's relative length difference. |
Tests
- Runsame tasks for all
- Testseach task's own tests
- Pass rateshare of trials passed
| Column | Meaning |
|---|---|
| Pass rate | The share of trials whose tests pass, counting errored trials as failures. |
| Cost / trial and p50 latency | The mean cost per trial at list prices and the median trial time. |
Tests, then votes
A family whose outputs are diffs, as a coding family's are, gives every system the same tasks, each a developer's request and a small repository in a sandbox without network access. Every system works on each task through the same agent within the same turn budget, which the board's Run step names, and tests the agent never sees then grade the solution.
The tests settle any pair they separate, so a passing solution beats a failing one without a vote. Two failing solutions tie without a vote, as do two passing solutions with identical diffs. The rater votes only between two passing solutions whose diffs differ, reading each diff without its commit history, as a reviewer reads a pull request.
Settled comparisons together count as at most one vote per task and system pair, with fractional weights when systems have several samples. Each human vote counts once, so the scores lean on the tests until the rater has voted on most pairs of passing solutions.
The rest of the board reads as Votes describes, apart from these terms:
| Term | Meaning |
|---|---|
| Failed tests | The share of its solutions that failed the task's hidden tests, and so lost their pairs without a vote. |
| Cost / solution and p50 latency | The mean cost of a solution at list prices and the median time of the agent's whole run on a task. |
| Diff length bias | How far the rater's votes lean toward the longer or the shorter diff, in score points per standard deviation of the pair's relative length difference. |
Known limits
- One person casts every vote, so each voted board reflects that person's preferences.
- Voted scores are relative, so only the both-bad and gate-failure rates measure outputs on an absolute scale.
- Voted scores carry no length adjustment. Each round summary reports whether the rater favors longer or shorter outputs.
The design notes on ranking, the vote arena, and each family cover the math.