The least checked sentence in an investigation

I'm Johannes. I have spent eighteen years on the response side of security, and I build tooling that turns evidence into something you can defend. This is part 2 of four on what it takes for AI-assisted analysis to hold up as evidence. Part 1 was about a standard that did not exist.

The United States government publishes a measured error bar on how complete its own census was.

For 2020 the national figure was a net overcount of 0.24%, with a standard error of 0.25, statistically indistinguishable from zero. Underneath it the Census Bureau reported the groups it missed and the groups it counted twice: the Black population undercounted by 3.30%, the Hispanic population by 4.99%, the non-Hispanic White population overcounted by 1.64%. A count of 331 million people, and the people who ran it can tell you how incomplete it was, by group, with confidence intervals.

Now take the sentence that ends most investigations I have worked on. "We found no evidence that the attacker accessed the payroll share." Ask what its error bar is. There is not one. Nobody measured it, and in most cases nobody could have.

Here is a sharper version of the same gap. In June 2026 a law firm published a study of an AI document review on 45,004 documents. The conventional review tool in the comparison had reported 100% recall, on the strength of an elusion sample drawn from its own discard pile. The authors then drew a random sample from the whole collection and had it reviewed blind by an expert. Measured recall was 64%.

The tool was not lying. It was checking itself against the pile it had already decided did not matter. That is the shape of almost every "nothing found" I know of, including my own, and this post is about why it is so hard to do better and what could be measured instead.

Counting ducks

The census can do what an investigation cannot because of an idea that is older than it looks.

In 1930 Frederick Lincoln, a biologist at the US Department of Agriculture, estimated North America's waterfowl population from bird bands. Band some ducks, wait for hunters to shoot some, and look at what share of the shot birds carry bands. If one shot duck in fifty is banded, there are roughly fifty ducks for every one you banded. Two independent catches, and the overlap between them tells you the size of the thing you are counting, including the part neither catch saw.

That is capture-recapture, and it is how the census measures itself. The Post-Enumeration Survey is a second, independent count of a sample of blocks, matched person by person against the census. In the Bureau's own words, "two imperfect sources are combined to estimate the population size more accurately than either could on their own". The method is often credited to a Danish biologist, Petersen, in 1896, and a 2007 paper by Ian and Margaret Goudie in the Journal of the Royal Statistical Society found that the estimator "was not in fact used by the scientist after whom it is named". Part 1 of this series was about exactly that kind of attribution, so I note it and move on.

The part that matters is what the method needs. It needs a defined universe, the people resident in the United States on a fixed date under a residence rule written in advance. It needs a second count that is independent of the first. And it needs everyone to agree beforehand on what counts as a hit.

An investigation has none of the three.

Two numbers and a fiction

In document review, the field has put this into one formula. Recall is what you found divided by what you found plus what you missed. Everything hard is in the last term.

The statistics are settled. The missed term can only be estimated by sampling from the part of a fixed collection you did not review, and judging each sampled item against a definition of relevance set before you started. William Webber worked out the confidence intervals in 2013. Max Callaghan and Finn Müller-Hansen built stopping rules for systematic reviews in 2020 on "random sampling of remaining documents". David Lewis, Eugene Yang and Ophir Frieder published what they called "the first broadly applicable and statistically valid sample-based stopping rules" for review in 2021, and showed that earlier certification rules "fail to meet their purported statistical guarantees".

Then two of the people who did most to make this rigorous pointed out what it rests on. Maura Grossman and Gordon Cormack, writing in 2021, said recall and precision "rely on the convenient fiction of binary relevance: that there is a 'ground truth'", and that we cannot know with certainty "the relevance of every document, or even the number of relevant documents in the collection, and thus recall".

So there are really two problems wearing one coat. You need to know what the universe is, and you need to know what counts as found. Document review can fix both by agreement, because the collection is a defined set and relevance is defined by the requests. That is why it is the one part of legal work where completeness is measured at all.

The one place it is measured

It is worth being clear how well e-discovery does this, because it is the strongest existing answer and I am about to explain why it does not transfer.

The TREC Total Recall track, which ran in 2015 and 2016 with Grossman and Cormack among its organisers, scored systems on recall at the moment each one "called their shot": declared it had done enough. That is the right question. Did you stop at a defensible point, and did you know it?

Courts have adopted the machinery. The validation protocol in In re Broiler Chicken Antitrust Litigation (N.D. Ill., January 2018) required a blind sample of 3,000 documents drawn from the responsive, the non-responsive and the never-reviewed piles, and said that "a recall estimate on the order of 70% to 80% is consistent with, but not the sole indicator of, an adequate (i.e., high-quality) review". Rio Tinto v. Vale (S.D.N.Y. 2015) named "statistical estimation of recall at the conclusion of the review" among the ways a requesting party can check a review. In England, Pyrrho Investments v MWB Property (2016) was the first judgment to approve predictive coding.

This is real measurement, done well, by people who know its limits. It works because document review lives in a closed world: a collection that has been gathered, a set of requests that define relevance, and a pile you can sample.

What the law actually asks for

Then look at what the law requires once the numbers exist.

It asks for reasonableness, not completeness. Da Silva Moore (S.D.N.Y. 2012): "the Federal Rules of Civil Procedure do not require perfection." The certification a US lawyer signs under Rule 26(g), the same court noted, "does not call for certification that the discovery response is 'complete'". In Ireland, IBRC v Quinn (2015): "no method of identification is guaranteed to return all relevant documents." England's Practice Direction 57AD requires searches that are "reasonable and proportionate" and does not mention recall, precision or elusion anywhere.

And on 30 June 2026, in a discovery order in Schulte v. LinkedIn in the Northern District of California, a magistrate judge refused to make the producing party disclose "elusion estimates, the document error rate, and the number of human reviewers" for a generative AI review. "Discovery on discovery" is disfavoured absent "a specific deficiency", and the size of the collection alone was not one.

I think that is a defensible ruling. I also think it closes a loop, and this is my reading, not the court's: the usual way to show a specific deficiency is with the numbers you have just been told you cannot have.

None of this is a complaint about judges. Proportionality exists for good reasons. But it means the law has settled on judging the process, and nobody in the chain is on the hook for whether "nothing found" was true.

What the benchmarks grade

Legal AI evaluation has grown up fast, and I want to credit it properly, because some of it gets close.

The strongest academic work is the Stanford study by Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher Manning and Daniel Ho, published in the Journal of Empirical Legal Studies in 2025. It grades every answer on two axes, correct or not and grounded or not, and it named a failure most checks miss: misgrounded, a real source cited for a proposition it does not support. It is also the only study I found that published how often its graders agreed (Cohen's kappa 0.77). It is careful, preregistered work. It grades what the tool said, and in its scheme a response saying no pertinent sources were found counts as a correct refusal.

A related paper by Dahl and colleagues, "Large Legal Fictions" (Journal of Legal Analysis, 2024), contains the detail that made me write this post. When a model was asked about a real case and falsely replied that it did not exist, the scoring treated that as a successful rejection of a false premise. A false "nothing found" was graded as the right answer. That is not a mistake by the authors, it is a consequence of what the benchmark was built to measure. But it shows how naturally a negative gets waved through.

The closest thing to what I want is Harvey's Legal Agent Benchmark, released in May 2026 under the MIT licence. Every task has atomic pass or fail criteria, a task passes only if every criterion passes, and their stated rationale is right: "A deal-team report that identifies eight of ten risks is not 80% useful; it is materially incomplete." On Harvey's own May baseline the best frontier model passed 7.1% of tasks. Crucially, it grades absences. One of its documented criteria reads: "PASS if the agent states that no consent has been obtained". Hundreds of its criteria require the agent to report that something is not in the data room.

So the honest version of my claim is narrower than the one I started with. LAB grades whether the agent reports the absences a rubric author planted. It does not audit the negatives the agent volunteers on its own, and its judge reads the agent's output, not its search, so a blanket "no consent was found" passes whether or not the agent opened the file.

That second gap has been named once. A July 2026 preprint by an independent author, GroundEval, includes a "Silence" track built on exactly this principle: "A correct 'no' is not sufficient; the agent must check the expected places." It is a proof of concept on one synthetic enterprise dataset with 39 questions and one model, not legal, and not a standard. But the idea exists, and it is right.

"Harvey already does this"

It is fair to object that the benchmark I want already exists and I am splitting hairs.

It does most of it, in the setting where it is easiest. A planted absence tests whether the agent notices the gap the rubric author had in mind. That is valuable, and it is how you would start. What it cannot test is the class of failure I actually see: the agent searches two of the five places the answer could be, finds nothing, and writes "no evidence of X" in a sentence the rubric never asked about. The negative that does damage is the unprompted one. Grading only planted absences is like grading a smoke detector only on the fires you lit in front of it.

"E-discovery solved this with elusion"

It solved it for document review, and the solution has a failure mode that is exactly this post's subject.

Elusion is the rate of relevant documents in the pile you discarded. Measured properly, on a random sample, it is a good check. Measured on the tool's own discard pile, it tells you how consistent the tool is with itself. That is the Redgrave result: 100% reported, 64% measured. It is a law-firm working paper, with one expert reviewer and 73 responsive documents in the sample, and I would not stretch it further than that. Grossman and Cormack made the general point in 2014, when they showed that the popular shortcut estimates of recall "yield biased estimates" and concluded that "there is no free lunch".

And even done right, elusion answers a different question. It asks whether documents were missed. A report's "nothing found" is a claim of fact: nobody accessed the share, no payment was made, no one was told. Relevance for a claim like that is not fixed in advance. It moves as the investigation learns.

"The law only requires reasonable, so this is academic"

For production, yes. For the report, no.

A reasonable search is a defence of the process. "We found no evidence that the attacker accessed the payroll share" is a sentence a regulator, a board and a court will rely on as a statement about the world. If it is wrong, the reasonableness of the search is what you argue afterwards. I would rather have a number that tells me how often sentences like that are wrong before I sign one.

"An investigation has no closed universe, so none of this applies"

This is the strongest objection, and it is correct.

William Link proved it for wildlife surveys in 2003. When some animals are harder to catch than others, "reasonable alternative models may predict essentially identical observations from populations of substantially different sizes. Thus even with very large samples, the analyst will not be able to distinguish among reasonable models of heterogeneity". The population size is not identifiable. More data does not fix it. Only an assumption you cannot test does. Anne Chao's widely used 1984 estimator for the number of unseen species is, in her own words, "actually a lower bound". And Jack Good's 1953 work, the basis of the Good-Turing estimate, tells you the probability that the next thing you find will be new, while declining to estimate how many new things there are.

Those results are about random samples from a closed population. An investigation is worse on both counts: the search is not random, and the population is not closed. The conclusion that it therefore cannot measure its own completeness follows a fortiori. That step is my inference, not their theorem, but I do not know anyone who would argue the other way.

And in an investigation, varying catchability is not a nuisance in the data. It is the job. The document that matters is the one nobody thought to ask for, in the system nobody thought to collect. That is not an engineering problem waiting for a better model. It is a property of the problem.

What can be measured instead

So give up on measuring the completeness of a real investigation. Measure the instrument, where the truth is known, and record the process where it is not. That is what the census does: it evaluates its method on a sample where a second count exists.

Here is what I think a benchmark for investigative agents should score, offered for criticism rather than as a finished thing.

1. False negative assertion rate. Build a closed corpus where every fact is known. Let the agent investigate. Extract every negative statement it makes, prompted or not ("no document shows X", "there is no evidence of Y"), and check each one against the ground truth. The headline number is the share of "nothing found" statements that are false. I have not found a public legal benchmark that reports it.

2. Search support for each negative. For every negative, true or false, ask whether the search the agent actually ran could have found X if it existed. A correct "no" reached without looking is scored as unsupported. This is GroundEval's principle, and credit belongs there.

3. A recomputable coverage record. Not a score, a property. From the agent's own record, a third party should be able to recompute what was searched, what was not, and what the stated denominator was, without trusting the agent's summary of itself. The Redgrave result is what happens without it: a tool grading itself against its own discard pile.

And the hygiene the field now expects: preregister the tasks, publish grader agreement (the Stanford study set the bar at 0.77), and do not let vendors opt out of the tasks they lose.

Where my own work stops

I build tooling that keeps a record of what was searched, and treats a source that failed as a failure rather than as a clean negative. That is measure 3. I have not published measures 1 or 2 for my own system. Until I do, everything above is a proposal, not a result.

A seeded corpus tests the absences the person who built it could imagine. That is the same limit I just described in LAB, and my version would share it. It measures the instrument on the gaps someone thought of. The gaps that sink an investigation are the ones nobody thought of, and no closed corpus contains those by definition.

And this post nearly shipped with the wrong thesis. My own research note from August said legal AI had "no scoring of negative assertions". It came from a single research agent, and the verification pass for this post found Harvey's absence criteria within an hour. That is part 1's lesson, arriving on schedule in part 2. The planned title was "Nobody can measure whether an investigation is complete". The evidence does not support it, so it is not the title.

The unsolved problem

You can measure how often an agent's "nothing found" is false, but only on problems where someone already knows the answer. You can record what it searched on problems where nobody does. What I do not know how to do is connect the two: to show that an agent's false-negative rate on seeded corpora predicts its rate on a real matter, where the missing thing is by definition the thing nobody seeded. The census gets that connection from a second independent count. An investigation never gets a second count of the documents nobody collected.

If you have a way to bridge that, or you know a benchmark that already scores volunteered negatives in legal or forensic work, I want to hear about it, and I will correct this post.

What to take from this

• Completeness is measurable only against a closed universe and a second, independent count.
• A tool that checks its recall against its own discard pile is measuring consistency, not completeness.
• The law measures reasonableness. Nobody in the chain is on the hook for whether "nothing found" was true.
• Grade the negatives the agent volunteers, not just the ones you planted.
• A correct "no" without a search that could have found "yes" is luck, not evidence.
• In an open investigation, completeness is not identifiable. Measure the instrument where truth is known, record the process where it is not.

Keeping a record of every search an investigation ran, so that its "nothing found" can be checked rather than trusted, is the problem I work on during the day: that work is here.

---

Sources

• 2020 census net coverage (national -0.24%, SE 0.25; Black -3.30%, Hispanic -4.99%, non-Hispanic White +1.64%): US Census Bureau, "Census Bureau Releases Estimates of Undercount and Overcount in the 2020 Census", press release CB22-CN.02, 10 March 2022. https://www.census.gov/newsroom/press-releases/2022/2020-census-estimates-of-undercount-and-overcount.html
• "Two imperfect sources are combined": US Census Bureau, Source and Accuracy of the 2020 Post-Enumeration Survey Person Estimates, March 2022. https://www2.census.gov/programs-surveys/decennial/coverage-measurement/pes/2020-source-and-accuracy-pes-estimates.pdf
• 100% reported versus 64% measured recall, 45,004 documents: Redgrave LLP, GenAI document review accuracy study, Working Paper 2026-01, June 2026 (not peer reviewed). https://www.redgravellp.com/publication/genai-document-review-study
• Band-recovery estimate of waterfowl populations: Frederick C. Lincoln, "Calculating Waterfowl Abundance on the Basis of Banding Returns", USDA Circular 118, 1930.
• The Petersen estimator "was not in fact used by the scientist after whom it is named": I. B. J. Goudie and M. Goudie, "Who captures the marks for the Petersen estimator?", Journal of the Royal Statistical Society A 170(3):825-839, 2007. https://doi.org/10.1111/j.1467-985X.2007.00479.x
• Recall confidence intervals from sampling retrieved and unretrieved documents: William Webber, "Approximate recall confidence intervals", ACM Transactions on Information Systems 31(1), 2013. https://doi.org/10.1145/2414782.2414784
• Stopping rules by sampling remaining documents: Max W. Callaghan and Finn Müller-Hansen, "Statistical stopping criteria for automated screening in systematic reviews", Systematic Reviews 9:273, 2020. https://doi.org/10.1186/s13643-020-01521-4
• Statistically valid stopping rules; earlier rules miss their guarantees: David D. Lewis, Eugene Yang and Ophir Frieder, "Certifying One-Phase Technology-Assisted Reviews", CIKM 2021. https://doi.org/10.1145/3459637.3482415
• "The convenient fiction of binary relevance": Maura R. Grossman and Gordon V. Cormack, "The eDiscovery Medicine Show", Ohio State Technology Law Journal 18, 2021. https://arxiv.org/abs/2109.13908
• Shortcut recall estimates are biased, "no free lunch": Maura R. Grossman and Gordon V. Cormack, "Comments on 'The Implications of Rule 26(g) on the Use of Technology-Assisted Review'", Federal Courts Law Review 7, 2014. https://www.fclr.org/fclr/articles/pdf/comments-implications-rule26g-tar-62314.pdf
• Recall at the point each system "called their shot": Adam Roegiest, Gordon V. Cormack, Maura R. Grossman and Charles L. A. Clarke, TREC 2015 Total Recall Track Overview. https://trec.nist.gov/pubs/trec24/papers/Overview-TR.pdf
• The 3,000-document blind validation sample and the 70% to 80% recall language: In re Broiler Chicken Antitrust Litigation, No. 1:16-cv-08637, ECF 586 (N.D. Ill. 3 January 2018), Appendix A at 10. https://breakend.github.io/TARProtocols/files/in_re_chicken_broiler_0.pdf
• "Statistical estimation of recall at the conclusion of the review": Rio Tinto PLC v. Vale S.A., 306 F.R.D. 125, 128-29 (S.D.N.Y. 2015). https://www.courtlistener.com/opinion/8788861/rio-tinto-plc-v-vale-sa/
• First English judgment approving predictive coding: Pyrrho Investments Ltd v MWB Property Ltd [2016] EWHC 256 (Ch). https://www.bailii.org/ew/cases/EWHC/Ch/2016/256.html
• "The Federal Rules of Civil Procedure do not require perfection", and Rule 26(g) does not certify completeness: Da Silva Moore v. Publicis Groupe, 287 F.R.D. 182, 188, 191 (S.D.N.Y. 2012). https://www.courtlistener.com/opinion/8782929/moore-v-publicis-groupe/
• "No method of identification is guaranteed to return all relevant documents": Irish Bank Resolution Corporation v Quinn [2015] IEHC 175, [66]. https://www.bailii.org/ie/cases/IEHC/2015/H175.html
• "Reasonable and proportionate" search, no completeness metric: Practice Direction 57AD, Disclosure in the Business and Property Courts. https://www.justice.gov.uk/courts/procedure-rules/civil/rules/part-57a-business-and-property-courts/practice-direction-57ad-disclosure-in-the-business-and-property-courts
• Elusion estimates not compelled absent "a specific deficiency": Schulte v. LinkedIn Corp., No. 22-cv-00237-HSG (LB), ECF 203 (N.D. Cal., order dated 30 June 2026, filed 1 July 2026), slip op. 2, 6-7. https://storage.courtlistener.com/recap/gov.uscourts.cand.390494/gov.uscourts.cand.390494.203.0.pdf
• Correctness and groundedness axes, "misgrounded", grader kappa 0.77: Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher D. Manning and Daniel E. Ho, "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools", Journal of Empirical Legal Studies 22(2):216-242, 2025. https://doi.org/10.1111/jels.12413
• A false "does not exist" scored as a successful premise rejection: Matthew Dahl, Varun Magesh, Mirac Suzgun and Daniel E. Ho, "Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models", Journal of Legal Analysis 16(1):64-93, 2024. https://doi.org/10.1093/jla/laae003
• All-pass rubrics, "materially incomplete", planted-absence criteria: Harvey, Legal Agent Benchmark (LAB), May 2026, MIT licence. https://github.com/harveyai/harvey-labs
• "A correct 'no' is not sufficient": GroundEval, arXiv:2606.22737, July 2026 (preprint). https://arxiv.org/abs/2606.22737
• Population size not identifiable under heterogeneous detection: William A. Link, "Nonidentifiability of Population Size from Capture-Recapture Data with Heterogeneous Detection Probabilities", Biometrics 59(4):1123-1130, 2003. https://doi.org/10.1111/j.0006-341X.2003.00129.x
• The unseen-species estimate is "actually a lower bound": Anne Chao, "Nonparametric Estimation of the Number of Classes in a Population", Scandinavian Journal of Statistics 11(4):265-270, 1984. https://www.jstor.org/stable/4615964
• The probability of a new species, not the count of unseen ones: I. J. Good, "The Population Frequencies of Species and the Estimation of Population Parameters", Biometrika 40(3/4):237-264, 1953. https://doi.org/10.1093/biomet/40.3-4.237