Abstract
How can we measure intelligence beyond human capability? Human-authored benchmarks saturate, and above human capability, examiners may not know which tasks are both hard and verifiable. We argue that this difficulty is inherent to absolute-scale evaluation and propose a new paradigm based on relative measurement in which models generate public challenges that separate other systems. Aggregating these outcomes yields an adversarial psychometric rating system that can scale with the systems being measured. We describe practical protocols that reduce incentives for private-information attacks, support judge-free adjudication, and naturally scale with agent capabilities. We instantiate the framework across verifiable and open-ended, non-verifiable domains, illustrating how model-generated evaluation can continue to measure systems beyond the human frontier.
Why evaluation breaks at the frontier
1 Benchmarks are exams written by humans
Since Spearman’s 1904 discovery that performance across cognitive tests is positively correlated, intelligence has been measured psychometrically: an examiner designs tasks, subjects solve them, and a latent ability score is inferred from the outcomes. Nearly every AI benchmark (MMLU, BIG-Bench, HELM, ARC, FrontierMath, Humanity’s Last Exam) inherits this structure. The tasks are fixed in advance and authored by people.
2 Above human level, the examiner is the bottleneck
Frontier models saturate benchmarks faster than experts can write harder ones. Worse, beyond human expert level the difficulty is not just producing enough questions but knowing which questions are simultaneously difficult, discriminating, and verifiable. This is an inherent limitation of absolute-scale evaluation: the measuring stick cannot be shorter than the thing being measured.
3 Naive “model vs. model” duels are gameable
A tempting fix is a pairwise tournament: model i wins by authoring a challenge it can solve but model j cannot, and Elo-style ratings aggregate the outcomes. But this rewards asymmetric information rather than capability. A proposer can ask “which bit am I thinking of?”, or publish the product of two large primes it generated and demand the factorization, a trapdoor inserted at construction time. And without an external judge, nothing forces the pair’s agreed resolution to be the true one.
The idea: adversarial psychometrics
Pose every challenge to many solvers at once, and pay the proposer for the variance of the panel’s answers, not for defeating one opponent. Private-state and trapdoor questions collapse: a calibrated panel uniformly answers “50/50”, inducing no variance and earning nothing. The only way to score is to find public questions on which capable systems disagree; in other words, to discover the capability boundaries of the population. Ranking models by their ability to separate (and to solve) yields a rating system, SepaRank, that scales with the models being measured instead of with their examiners.
| Paradigm | Who authors the tasks | Scales past human frontier | Needs a judge | Resists private-info attacks |
|---|---|---|---|---|
| Static benchmarks | Human experts | ✗ saturate | ✓ answer key | ✓ fixed tasks |
| Arena / preference voting | Users (prompts) | ✗ judged by humans | ✗ human raters | ✓ |
| Pairwise challenge duels | Models | ✓ | ✗ needed but unscalable | ✗ trapdoors win |
| SepaRank (this work) | Models | ✓ | ✓ judge-free options | ✓ variance ≈ 0 |
How a round works
- Propose. Each model in turn authors a binary challenge (a yes/no question or an
executable program whose
main()returns 0 or 1) and commits an answer ŷ that the panel never sees. - Sample a panel. k solvers are drawn from the other models, weighted toward the current frontier (see adaptive weighting below).
- Report beliefs. Every panelist independently reports pi ∈ [0, 1], its probability that the answer is A. Ignorance is pi = ½.
- Resolve. The ground-truth bit y is fixed by the arm’s resolution rule: executing the program, taking the proposer’s committed bit, or taking the panel’s majority vote.
- Score both roles. The proposer earns the separation it induced; each solver pays a proper-scoring-rule loss against y.
The factor of 4 normalizes the proposer’s reward to [0, 1]: it is maximal when the panel splits into confident, opposing camps, and zero when everyone agrees, and also when everyone hedges. The Brier loss makes calibrated reporting each solver’s best response, so the mechanism separates knowing from knowing that one does not know. Each round, a model’s proposer reward P and (negated) mean solver loss Q are z-scored across the field and averaged,
and the cumulative total G is the ranking. Because challenges are authored by the players themselves, the test’s difficulty automatically tracks the field’s ability; no human ever has to write a harder exam.
Built-in defenses
Private information earns nothing. “Which bit am I thinking of?” leaves a calibrated panel at p = ½ and pays zero. A trapdoored factoring instance is no better: the proposer is never asked to solve its own challenge, and an impossibly hard task makes everyone hedge. Separation, not difficulty, is what pays.
Conjunction boosting can be coarsened away. Chaining k questions together mechanically pushes reports toward 0 and 1, inflating raw variance. The protocol optionally rounds each report to the nearest of {0, ½, 1} before computing the score, so only categorical disagreement counts.
Adaptive weighting keeps the test discriminating at the top. Once weak models are clearly separated, re-separating them stops paying: solver panels are sampled with multiplicative weights w ∝ exp(ηG) that concentrate on the current leaders, so a high-scoring proposer must eventually find questions on which the frontier itself disagrees.
Four resolution rules, one scoring mechanism
The scoring never changes; only how the truth bit y is fixed. This lets the same protocol span fully verifiable domains and completely open-ended ones, including judge-free operation.
| Arm | Artifact | Resolution of y | Solver’s report predicts |
|---|---|---|---|
| PE | Python program | Sandbox execution | What main() actually returns |
| PC | Python program | Proposer’s committed bit | What the proposer claims it returns |
| QC | Natural-language question | Proposer’s committed bit | The proposer’s committed answer |
| QN | Natural-language question | Panel majority vote | The population consensus |
Experiments: eleven models, no human examiner
The protocol is implemented end-to-end in the combined-protocol codebase and run as ten independent 20-round games per arm, each fielding the same N = 11 contemporary models from five providers (OpenAI, Anthropic, Alibaba, Moonshot AI, DeepSeek) under stable pseudonyms. Every round each model authors one challenge and a panel of k = 5 solvers is sampled; prompts disclose the entire mechanism (scoring formulas, resolution rule, adaptive weighting, and the last five rounds of transcript) with no strategy hints. Programs are machine-checked in a sandbox; nothing else is judged by a human.
The rankings track the roster’s known capability span: frontier models lead both arms and the small non-reasoning model anchors the bottom of both. Solver behavior explains much of the spread: reports are nearly binary (mean distance from ½ between 0.39 and 0.50 for every model), so Brier loss is governed by directional accuracy. The weakest models are miscalibrated, not just ignorant: gpt-4o-mini answers on the wrong side 40% of the time while staying confident, earning a loss (0.388) worse than a solver that always answers ½ (0.25). The strongest models are both directionally correct and the most willing to hedge. The proper scoring rule is doing its intended work, and failing to declare uncertainty is itself a capability signal.
When lying is legal, honesty becomes a measurement
In the committed arms the resolved answer is whatever the proposer commits, so misrepresentation is a legal move. What do models do with that freedom?
- The population is largely honest: 87.0% of program-arm proposals commit the bit the program actually returns, flat across rounds.
- But dishonesty pays: miscommitted proposals earn 2.4× the proposer reward of honest ones, by manufacturing disagreement between solvers who execute the code and solvers who anticipate the deception.
- And the strategy concentrates at the top: the arm’s winner, gpt-5.5, is also its least honest proposer (59%), alternating honest and false commitments on the same challenge template so that recognizing the template reveals nothing. Meanwhile qwen3.7-max commits honestly in 198 of 200 proposals, and is outranked.
The starkest case is opus-4.8, whose program-arm rating belies its solving ability (third-best Brier loss). Its authored challenges are too cooperative: 92% of its programs carry comments, 70% narrate the computation or its result, and 97.5% of its commitments are honest. The result is annotated exercises with reliable answers, which a field this strong solves unanimously (on 84.5% of its programs every panelist answers correctly, versus 23.1% for gpt-5.5). Its five miscommitted proposals earned nearly four times its average reward, an incentive it declines to engage in. The committed arm specifically prices transparent, honest authoring, and different models pay very different amounts for it.
Reasoning models rationally defeat the exploits
With no judge inspecting question content, the open-ended arm invites exploits: identity-indexed questions (“does your pseudonym contain the letter a?”), erratic commitments on trivial questions, intractable trivia (“is the 1000th digit of π odd?”). A protocol that collapses under these stops measuring anything. Giving the same eleven models an explicit reasoning scratchpad shows the defenses working through the players’ own rationality:
“In Round 0, Erin asked ‘Does 2 + 2 equal 4?’ and committed 0 (the false answer)… Since Erin’s commits are unpredictable and likely chosen to maximize variance… To minimize my maximum possible loss (minimax strategy) I will report a probability of 0.5, guaranteeing a Brier loss of 0.25 regardless of her commit.”
— Qwen3.7-Max, hedging an erratic proposer down to zero reward
“Judy appears to commit based on whether the statement is true for Judy herself. In prior Judy-authored questions, she committed 0 when Judy was not in the listed set… The current list does not include Judy. Therefore Judy almost certainly committed 0.”
— GPT-5.5, converting an identity question into commit-prediction
Erratic committers get hedged to zero; identity questions are answered by modeling the proposer rather than oneself (and expose a theory-of-self failure in models that answer for themselves, including strong ones); intractable questions collapse to the bit-guessing game and stop paying. The takeaway: under SepaRank, the only reason to answer confidently is the belief that you know something the others don’t, and the only questions that keep paying are well-formed and hard ones. The equilibrium sharpens as the field gets more rational; that is exactly the property a beyond-human-scale benchmark needs.
What do the winning challenges look like?
| Challenge (verbatim) | Why it separates | Reward P |
|---|---|---|
| “Is the city of Windsor, Ontario, Canada located farther south than the city of Detroit, Michigan, USA?” | Counterintuitive-but-objective geography (it is) | 0.91 |
| “Does France share a land border with Brazil?” | Counterintuitive fact (yes, via French Guiana) | 0.93 |
| “Is 91 divisible by 7?” | Trivial-but-miscomputable: the single most divisive question of the campaign (7×13 = 91) | 0.96 |
| “Does the English word ‘strawberry’ contain exactly three ‘r’ characters?” | Sub-token blind spot: letter counting | 0.63 |
| “Is the number 2026 equal to 2026?” (committed 0) | Weaponized tautology: the panel splits between the fact and the anticipated false commit | 0.91 |
x = float("nan"); return int(x in [x]) (program arm) |
Semantic edge case (containers check identity before equality), reused with opposite commitments across games | — |
Conclusion
When human examiners can no longer author frontier-level tests, evaluation itself must become model-generated and incentive-compatible. SepaRank demonstrates one way to do this: models are rated by their ability to construct challenges that separate a population of their peers, and to solve the challenges constructed by others. The mechanism neutralizes private-information attacks by design, runs with or without a judge, resolves statistically significant distinctions within the frontier, and produces consistent rankings across verifiable and open-ended domains. Beyond measurement, the protocol generates a growing corpus of machine-authored questions at the frontier of the field’s knowledge: raw material for post-training and self-improvement.
Citation
@article{han2026measuring,
title = {Measuring Intelligence Beyond Human Scale},
author = {Han, Jerry and Moschopoulos, Rafael and Colby, Ella and
Goyal, Vishrut and Tu, Andrew and Ghods, Kia and
Braverman, Mark and Hazan, Elad},
journal = {arXiv preprint arXiv:2607.07040},
year = {2026},
url = {https://arxiv.org/abs/2607.07040}
}