A benchmark from The Axiom Foundation
Can a model write the law as code?
EncodeBench measures how well AI models turn statutes into cited, executable rules. Every score comes from deterministic gates: the encoding compiles, passes CI, and contains no number the source text doesn’t.
Its sister benchmark, PolicyBench, asks whether a model can compute the law. EncodeBench asks whether it can write it.
The board — UK v1
Run v3 · axiom-encode 0.2.138216 cases, six models, every run bound to the signed corpus release uk-rulespec-2026-07-14. Headline is the deterministic gate: the encode succeeds, the RuleSpec compiles, its companion tests pass, and no numeric literal is ungrounded.
| runner | model | gate pass | T | artifacts | compile | ci | grounded | median |
|---|---|---|---|---|---|---|---|---|
| gpt-5.5 | gpt-5.5 | 15/16 93.8% | — | 16/16 | 100% | 93.8% | 100% | 48s |
| sol | gpt-5.6-sol | 14/16 87.5% | — | 16/16 | 100% | 87.5% | 100% | 45s |
| fable | claude-fable-5 | 12/16 75.0% | 1 | 15/16 | 100% | 80.0% | 100% | 266s |
| terra | gpt-5.6-terra | 11/16 68.8% | — | 16/16 | 100% | 68.8% | 100% | 34s |
| luna | gpt-5.6-luna | 8/16 50.0% | — | 15/16 | 93.3% | 53.3% | 100% | 50s |
| opus-5 | claude-opus-5 | 6/16 37.5% | — | 16/16 | 87.5% | 37.5% | 100% | 47s |
Every number here came out of a signed results.json through the fold — none is hand-entered. A T is a case that hit the harness time ceiling, not a model that failed: those cases are held out of the compile and grounding denominators rather than counted as errors.
This run is not reasoning-effort matched. The harness never set it. Its OpenAI path passes -c reasoning_effort, which is not a recognized Codex config field — Codex accepts the flag and silently ignores it — and its Anthropic path passes no effort flag at all. Every runner therefore used its own CLI default, unset and unrecorded. A follow-up audit also found the backends did not sit the same exam: the Codex runners could read workspace files while the Anthropic runners ran prompt-only. Cross-family rows here should be read loosely. The next run fixes both — explicit recorded effort and an identical prompt-only exam for every backend.
Four gates, one headline
A case passes for a model only when all four deterministic checks pass. The headline metric is the gate-pass rate and nothing else folds into it.
- 1 · encode
The model receives one provision's text, resolved from a signed corpus release, in a cold workspace: the source, stubs for defined terms, and no existing encoding of the target. It writes the RuleSpec module.
- 2 · compile
The module compiles in the Axiom rules engine.
- 3 · ci
Repository validation passes, including the companion tests every rule must carry.
- 4 · grounded
Zero ungrounded numeric literals: every number in the encoding appears in the source text. A model that recalls a rate instead of reading it fails here.
Reported alongside, never folded in: a statutory-fidelity review score, how many of the statute’s numbers the encoding captured, a PolicyEngine oracle check where wired, latency, and cost.
The suite — 16 cases, stratified
UK tax and benefit law, chosen to span the skills encoding demands. Every case resolves from the same signed release, so every model reads the same words.
Parameters and thresholds
Flat rates and fixed amounts — the floor. A model that invents a rate here fails the grounding gate, not a reviewer's judgment.
| # | provision | what it probes | workspace |
|---|---|---|---|
| 01 | Value Added Tax Act 1994 s. 2 uk/statute/ukpga/1994/23/2 | Flat-rate control case | cold |
| 02 | Social Security Contributions and Benefits Act 1992 s. 13 uk/statute/ukpga/1992/4/13 | Weekly flat amount plus a small-profits threshold | cold |
| 03 | Social Security Contributions and Benefits Act 1992 s. 9 uk/statute/ukpga/1992/4/9 | Employer-side rate and secondary threshold | cold |
Bracket and band structure
Rate tables and band-conditional amounts.
| # | provision | what it probes | workspace |
|---|---|---|---|
| 04 | Income Tax Act 2007 s. 10 uk/statute/ukpga/2007/3/10 | Basic, higher, and additional rate bands | cold |
| 05 | Income Tax Act 2007 s. 13A uk/statute/ukpga/2007/3/13A | Band-conditional nil-rate amount | cold |
Phaseouts and tapers
Where encodings usually go wrong: interacting thresholds, withdrawal rates, and rounding directions.
| # | provision | what it probes | workspace |
|---|---|---|---|
| 06 | Income Tax Act 2007 s. 35 uk/statute/ukpga/2007/3/35 | £100,000 half-excess taper with round-up | cold |
| 07 | Income Tax (Earnings and Pensions) Act 2003 s. 681B uk/statute/ukpga/2003/1/681B | High income child benefit charge: £200 per £100 over the threshold | cold |
| 08 | Finance Act 2004 s. 228 uk/statute/ukpga/2004/12/228 | Annual allowance beside the s. 228ZA taper | cold |
Cross-person and structural mechanics
Elections, entity scoping, and multi-step formulas.
| # | provision | what it probes | workspace |
|---|---|---|---|
| 09 | Income Tax Act 2007 s. 55B uk/statute/ukpga/2007/3/55B | Transferable allowance: elections and both-party conditions | cold |
| 10 | Income Tax Act 2007 s. 23 uk/statute/ukpga/2007/3/23 | The multi-step liability calculation; import-heavy | cold |
| 11 | Welfare Reform Act 2012 s. 8 uk/statute/ukpga/2012/5/8 | Universal credit award: maximum amount minus income reductions | cold |
Grounding discipline
Cases built to catch invented numbers. One provision whose amounts live elsewhere; one act too new to be in any training set.
| # | provision | what it probes | workspace |
|---|---|---|---|
| 12 | Social Security Contributions and Benefits Act 1992 s. 141 uk/statute/ukpga/1992/4/141 | Entitlement conditions only — the weekly rates live in the 2006 regulations, so any rate literal here is fabricated | cold |
| 13 | 2026 c. 11 s. 1 uk/statute/ukpga/2026/11/1 | Recency probe: the income tax charge for 2026–27, postdating model training data — the encoding must come from the supplied text | cold |
Oracle candidates
Provisions on the PolicyEngine UK comparison path. They run with repository context and will carry a live oracle check once that plumbing is verified; until then they grade on the same deterministic gates.
| # | provision | what it probes | workspace |
|---|---|---|---|
| 14 | Income Tax Act 2007 s. 6 uk/statute/ukpga/2007/3/6 | Main income tax rates on the worker oracle path | repo-augmented |
| 15 | Social Security Contributions and Benefits Act 1992 s. 8 uk/statute/ukpga/1992/4/8 | Employee-side Class 1 contributions on the worker oracle path | repo-augmented |
| 16 | Child Benefit (General) Regulations 2006 reg. 2 uk/regulation/uksi/2006/965/2 | The enhanced and other weekly rates — the regulation the entitlement case must not invent | repo-augmented |
Methodology
- Cold workspaces
- With repository context, a model can lean on — or copy — the encoding that already exists for a provision, and scores drift as the repository grows. Capability cases run cold: the same source text and nothing else, identically for every model.
- Boards refuse to blend
- Runs fold into one board only when the case set, the corpus release, and the score-affecting toolchain identity all match; checkout locations are ignored, everything else must be byte-identical. Add a model later without re-running the rest. Change a case, the release, or the encoder, and it's a new board — by construction, not convention.
- The reviewer never scores the headline
- A pinned model reviews each encoding's statutory fidelity, identically for every contestant. It is one judge, it is noisy, and when its own family is on the board it is not a neutral party — so its score is reported next to the gates and never folded into them.
- A control model rides along
- The roster includes a model that prior internal measurement found weak at this task. If the board can't separate it from the leaders, the instrument — not the models — is under suspicion.
- Every run carries its provenance
- Each run binds the exact corpus release (Ed25519-verified), the encoder and rules-engine versions, and the repository and waiver state that graded it, with signed result evidence. A cell on the board traces to precisely what produced it.
EncodeBench formalizes the internal encoder bake-off (July 2026) that routes The Axiom Foundation’s own encoding pipeline. The suite, harness, and fold live in axiom-encode (PR #1191).