Extending Legal Agent Bench to U.S. Case Law

Today we introduce LAB: United States Case Law (LAB: USCL), an evaluation framework for testing frontier models’ ability to research and draft litigation briefs end-to-end using real case records. LAB: USCL adds novel approaches to LAB for understanding model behavior by building environments from real, public data collected by Free Law Project and evaluating model-written briefs head-to-head.

We find that a small group of frontier models can now research and write complete, first-draft briefs in a single shot with no attorney guidance. While these briefs still exhibit a long tail of failure modes, they are substantially closer to the high bar for filing with a court and mark a substantial advance on a task that demands complex legal reasoning and advocacy.

In this post we discuss our initial findings on these emerging capabilities as well as how we plan to make research in this area more accessible in partnership with Grubel AI.

Brief Writing: An Ultra-Long Horizon Task

When parties to a litigation want a court to act, they file a motion: a formal request asking the judge to issue a specific order or ruling. A motion may ask the court to do routine things like change a deadline or take more dramatic actions like dismissing a case or sanctioning a party. Some motions are opposed by the other parties, who argue that it would be legally incorrect or otherwise inappropriate for the court to take the requested action.

To support these more contested motions, parties file briefs: written advocacy explaining why the court is justified in taking some action based on the state of the case, the law and precedent, and other considerations. Brief writing is one of the most complex tasks in litigation, sometimes requiring a team of lawyers multiple weeks to write and refine a compelling argument. An agent replicating this work must complete the same three steps a lawyer does: understanding the state of the case, researching the case law, and writing the brief itself.

Understanding the State of the Case. For agents, the first part of replicating this work is understanding the state of the case by reviewing its docket, the official record of the case’s proceedings. The docket lists every filing, meaning every document submitted to the court, such as the complaint that starts the lawsuit, the motions and briefs that contain the parties’ arguments, and exhibits supporting those arguments, such as contracts, letters, or financial records. It also records the judge’s orders and rulings. In complex or long-running cases, the docket can span thousands of filings.

Researching Case Law. The next step is case law research: finding the past court decisions that bear on the legal questions in a case and determining how much weight each one carries. When judges rule on motions or decide cases, they often explain their reasoning in written opinions. Together, these opinions make up case law, the body of rules that comes from judges’ decisions, as opposed to rules passed by legislatures (statutes) or issued by agencies (regulations). These decisions carry weight because they create precedent. In the United States legal system, judges are expected to justify their decisions by relying on earlier opinions. Some of those opinions are binding: a court must follow decisions on the same legal question from higher courts in its jurisdiction. Other opinions, such as decisions from other trial courts, are only persuasive, so a court can look to them for guidance but doesn’t have to follow them.

Case law research requires an agent to not only find the relevant opinions but also read them against the broader body of precedent, because later decisions can narrow, modify, or even invalidate an earlier opinion. Read in isolation, an opinion may seem to support a client’s position even after a higher court has rejected it. A brief that relies on that opinion misstates the law and weakens the lawyer’s credibility with the court. In the figure below, for example, a brief relying on the Fifth Circuit’s test would be citing a rule that is no longer valid following the Supreme Court’s ruling.

Writing the Brief and the Bar for Filing. Once an agent has researched the facts and the law, the final step is writing the brief itself. A strong brief must synthesize that research and connect the facts of the current case to precedent to persuade the court to rule in the client’s favor.

An agent performing this task must keep the details of hundreds of pages of filings and opinions straight, strategize which arguments to make and which to leave out, explain why favorable cases control and unfavorable ones don’t, and turn all of that into a persuasive document that follows strict conventions on length and structure, with every citation to the record and case law accurate.

The bar for that final document is particularly high. Unlike a research memo or other internal analysis, a brief is filed on a public docket on the client’s behalf. The attorney who signs it certifies to the court that existing law supports its legal arguments and all factual claims have evidentiary support. Lawyers also owe the court a duty of candor. They cannot misstate the law, and they must disclose directly adverse controlling authority. Mistakes have consequences beyond simply a weaker argument. Lawyers can be sanctioned, the client can be held to the brief’s positions, and an argument left out may be lost for good, including on appeal. A brief is ready to file only when every citation, characterization, and strategic choice can withstand that scrutiny.

Building Case Law Environments from Real Data

LAB: USCL measures agents’ capabilities to perform this task against substantially the same materials a lawyer used in a real case. We select a brief actually filed in court, then ask the model to carry out the same assignment: writing a brief for the same party, addressing the same motion or legal issue at the same point in the case.

To build these environments, we sample from federal court dockets. In the United States, dockets are generally public and accessible through PACER, the federal judiciary’s electronic records system. PACER holds the dockets of more than 200 federal courts, covering millions of cases and more than a billion documents. Because PACER charges per page for access, our pipeline draws from RECAP, Free Law Project’s community-constructed archive of PACER filings. RECAP is built from documents that PACER users have already purchased and shared, so its coverage can be uneven. For example, a docket may list a brief or exhibit that no one has downloaded and so is unavailable on RECAP.

We therefore filter for dockets that contain (1) a filed brief and (2) the filings necessary to contextualize that brief to ensure it can support a realistic environment. Even then, a docket isn’t a comprehensive record of the case. Most discovery, such as deposition transcripts and other witness testimony, is never filed with the court unless a party attaches it to a motion, and sensitive material may be sealed or redacted. The resulting environment gives the model enough to understand the case, but not everything the lawyer had.

For each environment, we then extract all case citations from the filed brief. We use agents to locate the cited judicial opinions in CourtListener, Free Law Project’s case database of more than 8 million opinions from across the judiciary. We retain an environment only if we can match more than 95% of the case citations in the lawyer’s filed brief to opinions available in CourtListener. This ensures the model works from substantially the same case law the lawyer used to draft the filed brief.

Even with these filters, the scale of the archive yields a large pool of valid environments. Building them, however, is not straightforward. Gaps in the docket mean an automated pipeline often can’t tell whether a case record is complete enough to support the task. But because one docket can contain multiple briefs, a single docket may yield several environments. From an initial sample of 200 dockets, our pipeline’s first pass produced 40 environments. Through human review and manual correction, we expanded that to a final set of 100.

To ensure diverse coverage, we taxonomize briefs across a number of salient features, such as court type, filing posture, and case type. The initial set spans trial and appellate courts, every stage of motion briefing, and more than ten areas of law. As we expand our environments, there is substantial ground to cover to ensure we can capture the full diversity of potential jurisdictions, motions, and legal disputes.

Evaluating Agents Head to Head

For each environment, we define two task conditions: open- and closed-book. In the closed-book condition, the model receives a file system, built by our extraction pipeline, containing the case’s record and the cases the filed brief relied upon. In open-book, the model receives only the subset of filings relevant to defining the task, along with access to legal research tools through a CourtListener MCP to search for relevant case law and additional filings, limited to materials available before the brief was filed.

In both conditions, a model is given a single short instruction to draft the relevant brief. From there, it works on its own, with no attorney feedback or follow-up prompts. Both conditions use the standard LAB harness with compaction to handle long context and OCR tools added to handle PDFs.

Rather than grading each brief against a fixed answer key, we have models’ briefs face off directly. We evaluate model-written briefs pairwise in a round-robin tournament using GRMs which select the better brief (or declare a tie) and provide reasoned criteria supporting each choice. These criteria are reviewed by lawyers and calibrated to ensure they are relevant to reasons expert reviewers would prefer a brief.

We then aggregate a model’s wins and losses in these head-to-head games into an Elo rating for each model. This rating predicts a model’s relative brief-writing skill, allowing us to order models based on how well they write briefs and predict how likely they are to write a winning brief outside of our sampled environments.

Initial Findings: A Small Group of Frontier Models Are Writing Complete, First-Draft Briefs

The pairwise results substantially separate model performance, with nearly 700 Elo points separating the highest- and lowest-rated models. This means that the highest-rated model, Opus 5.5, writes a stronger brief than the lowest performing models more than 95% of the time.

This performance is stable across open- and closed-book environments, suggesting model performance does not come from their ability to effectively use search tools, but their ability to convert the materials into complete and structured briefs.

As we look at the highest scoring models, the “Frontier Writers,” we find not only relative improvement but an emerging capability to write briefs end-to-end. These models are not only doing better research and advocacy, they are incorporating the depth and craft required to produce a polished work product.

The below video illustrates these gains. It compares a brief produced by a recent frontier model (left) with one produced by its immediate predecessor in the same model family, released just a few months earlier (right). The newer model’s brief is longer, engages with each of the opposing party’s arguments, and follows the conventions of a filed brief, down to its cover page and table of authorities. Its predecessor’s brief is substantially shorter and missing the hallmarks of a correctly formatted brief.

These improvements change the evaluative methods needed to assess the quality of model-written briefs. Previous evaluations found that while models were getting better, they were far from good, with the question of “is this a brief” answerable in seconds from responses that were far too short and lacked the substance and structure of a filed brief. As models increasingly pass these hurdles, deeper analysis is required to validate and improve their capabilities.

Under that closer scrutiny, even the strongest briefs still fall short of the filing bar and show a long tail of failures. In addition to longstanding issues like hallucinations, leading models unnecessarily concede points that weaken the client’s position or abandon strong arguments made in the party’s initial brief. They may rely on a court decision that has been set aside, or defer to a decision the court is not required to follow, without considering why that decision may not apply. These errors can appear within otherwise sophisticated arguments or may occur only in some cases as models display jagged intelligence across two similar tasks.

What’s Next

These findings define two connected research priorities: improving models’ ability to produce accurate, strategically sound briefs, and developing reliable ways for experts to supervise and evaluate that work at scale.

To support this research, we plan to open source resources in partnership with Grubel AI, enabling researchers to construct their own case law environments and study model capabilities. We want to provide people with cookbooks to build the environments they’re interested in, whether for a specific court or type of litigation, or for courts not on PACER, such as state courts or the Supreme Court.

Unlike previous LAB releases, we are not currently publishing sample environments alongside this post. These environments draw on real litigation involving identifiable people and organizations. Although the filings are public, we want to avoid drawing additional attention to selected disputes and the allegations they contain.

Until recently, even the best models struggled to write briefs without close attorney supervision and days of iteration. Today, leading models are beginning to research and write compelling briefs in a single pass, marking a substantial advance in their ability to carry out complex legal work. LAB: USCL provides a way to measure that progress through environments built from the same sources of data litigators use every day.

We’re excited to deepen this research and give the community the tools to do the same.