◆ Backtest methodology
How the multi-persona team gets audited against the Texas appellate record · published, not benchmarked
We pick a Texas appellate case the losing side already lost. We strip the outcome from the record, drop the team into the losing party's shoes mid-litigation, and let them argue from scratch — without knowing how it ended. Then we score the argument they produce against what actually won.
The category gap
Legal AI tools today produce confident output and never get tested against the historical record. Nobody publishes a number on how often the system produces a better argument than what was actually filed. Backtesting closes that gap. We take Texas cases that already happened — where the winner and loser are known — and run the deliberation team on the losing party's facts, blind to outcome.
This is not a benchmark designed to look good. It is an audit of our own quality, published with the methodology open for inspection.
Case selection
Cases enter the backtest pool when three conditions hold:
- Loss reason is doctrinally clear. “Petitioner failed to address §X” or “brief did not cite the controlling holding in Y” are research-quality questions. Credibility findings and jury-discretion losses are excluded.
- Losing party's facts are recoverable from the opinion. Most Texas appellate opinions describe both sides' positions in the background section; some are too sparse for blind replay.
- Current doctrine still applies. Cases turning on doctrines that have since evolved are excluded so the backtest reflects today's landscape.
The pool is bucketed across practice areas — civil litigation, family, real property, criminal, and business commercial — so the published rate isn't skewed by a single area's peculiarities.
The run + the rubric
For each backtest case, the orchestrator feeds the deliberation team the losing-party facts and procedural posture — blind to outcome. The team runs its usual iterative review: named personas debate, citations are cross-examined by the citation auditor, the adverse-authority persona surfaces controlling cases against the team's own conclusions, and the synthesis is produced.
The synthesis is then scored against the historical record on four dimensions:
- Doctrinal accuracy — did the team identify the controlling rule?
- Authority quality — were the cited cases on point and good law?
- Adverse-authority handling — did the team confront the cases that ultimately decided the matter?
- Argument structure — would the synthesis have plausibly altered the outcome?
A composite score of 4 or 5 on the four-dimension rubric counts as a “would-have-won” result. Scores of 1 to 3 count as no change from the actual outcome.
Blinding
The single field that matters for honesty is correctOutcome — the actual disposition of the case. It is set on the dataset record at curation time and is never shown to the deliberation team during a backtest run. It is used only when the scoring step compares the team's synthesis against the historical record. The blinding is enforced at the orchestrator level, not by convention.
What this is not
This page is informational, not legal advice. The deliberation team performs research review; it does not practice law and is not a substitute for retained counsel. The published backtest number is a measure of the team's research quality against the historical record — not a guarantee of any outcome in any specific matter.