Law as a Verifiable Domain: Using Real-World Judicial Outcomes to Evaluate AI Legal Reasoning

Reinforcement learning (RL) techniques, in domains that can offer objectively verifiable rewards, have led to incredible and noteworthy recent gains in the capabilities of frontier AI models. But not all domains are equally verifiable. Many high-level legal judgment tasks are difficult to verify, and model capability gains on those tasks have not been nearly as noteworthy as model gains in domains that are susceptible to automated verification like math and coding.

This paper argues that the heavy reliance of existing legal benchmarks on human-written rubrics and artificially constructed environments fails to take advantage of key properties of the common law judicial tradition, which gives precedential effect and the force of law to reasoned judicial decisions issued by judges in specific cases or controversies. For AI researchers, those judicial dispositions supply an external signal that allows for evaluation of some high-level legal judgment tasks against an objectively verifiable real-world ground truth. In the legal system, the opinions of legal experts or data providers as to what the law is have no authoritative or precedential value. The pronouncements of a tribunal, by contrast, carry the force of law (and often precedential effect).

LegalForecastBench illustrates one mechanism by which the signal from judicial decisions can be used to objectively evaluate the legal reasoning capabilities of frontier models. Models receive the same briefs that a federal judge received when ruling on a motion to dismiss, and are asked to estimate the probability that the judge dismissed each claim challenged by a defendant.

The beta includes 91 cases and 387 scored claim–defendant units. Of the 18 frontier models tested, GPT-6 Sol has the lowest micro-Brier loss (0.118) and the highest accuracy (84.5%). Notably, calibration had a large impact on the performance of models. Claude Fable 5.1 had no errors in its 81 high-confidence (at least 90%) forecasts, while for Gemini 3.1 Pro Preview, 57 out of 320 high-confidence predictions it made were inaccurate. Several newer model versions scored below their predecessors on this cohort, although none of these differences was statistically significant where it could be tested.