A search on 5 September 2026 went looking for what is known about a model finding, in a transcript, the evidence a coding instruction asks for. The results are worse for evidence-based designs than we expected, in ways worth writing down.

Agreement on codebook-driven coding can be decent and does not transfer. Dunivin (2024, arXiv 2401.15170) found GPT-4 at Cohen's kappa of 0.6 or better on eight of nine socio-historical codes, where GPT-3.5 on identical prompts managed a mean of 0.34. Balt and colleagues (2025, Frontiers in Public Health 13:1512537) got 0.84 mean accuracy and kappa 0.68 from an offline Llama 3 70B against consensus human coding of 2,666 segments from 38 interviews. Xiao and colleagues (IUI '23 Companion, arXiv 2304.10548) got kappa 0.61 on a binary dimension and 0.38 on a four-category one, where their human experts agreed at 0.88 and 0.90 on the same data. A scoping review of 75 studies reports agreement anywhere from 36 to 99 per cent (Kempny and colleagues, BMC Medical Research Methodology 26:137, 2026). So kappa in the 0.6 to 0.8 band is reachable on a well-specified codebook, is not guaranteed, and cannot be borrowed from somebody else's paper.

The failures are structured. Two results matter more than the headline numbers.

• Locating is much harder than classifying. Balt and colleagues report 0.84 accuracy when the model classifies a segment it is handed, and 0.67 when it classifies the transcript itself, window by window, rather than being handed the coded segments. Most coding for evaluation is the second kind: read nine thousand words and find the passages bearing on a proposition. So 0.67 is the relevant number, and the 0.84 that gets quoted in AI-for-qualitative-research talks is not.
• Aggregate accuracy hides constructs the model is not measuring at all. Pangakis, Wolken and Fasching (arXiv 2306.00176) replicated 27 annotation tasks from recent social-science articles: median accuracy 0.850, and yet nine of the 27 had precision or recall below 0.5. Chew and colleagues (RTI International, arXiv 2306.14924) found human-model agreement above 0.76 on most codes and 0.18, 0.36, 0.50 and 0.58 on four of them, where the human coders agreed with each other at 0.96, 0.93, 0.79 and 1.00 on those same four. The model failed hardest exactly where the humans found the task easiest.

For any study that compares theories or groups, that last pattern is the danger. A theory whose evidence happens to sit in the codes a model handles badly loses, and the loss is invisible in an aggregate figure.

Two devices from this literature are worth adopting.

• A test of randomness that uses no human labels. Chew and colleagues formalise a check on the model's own output distribution, a two-sided binomial for binary codes and a chi-squared against an equal-probability multinomial for categorical ones, reasoning that a model that has not grasped a construct is reduced to assigning codes at random. On their data the four codes that failed this test were exactly the four where agreement with humans later collapsed. That is a diagnostic that can run over a coding's output before anybody looks at the results, and a natural pre-registered gate. It has a known blind spot, a false negative when the true base rate sits near half.
• Repeated sampling, where consistency tracks correctness. Pangakis and colleagues classify each sample at least seven times and take the modal label; accuracy runs 19.4 percentage points higher on the samples where all runs agreed. One caution the authors carry themselves: 85 per cent of their samples were fully consistent, so most errors sit inside the consistent group, and consistent must never be read as correct.

Cohen's kappa is the wrong instrument for rare codes. Chew and colleagues report Gwet's AC1 instead, because many of their codes are rare and kappa "can suffer in these settings" from the high-agreement, low-reliability paradox. Smoking-gun evidence in process tracing is rare by construction, so a reliability figure on a sufficiency test should be AC1 or an explicit likelihood ratio, and never a kappa.

The judge step has its own known biases. Zheng and colleagues (NeurIPS 2023 Datasets and Benchmarks, arXiv 2306.05685) name four in LLM-as-judge: position, verbosity, self-enhancement and limited reasoning, and say themselves that they mitigate only some of them. Verbosity bites wherever one theory's tests happen to surface longer passages. The cheap mitigation is to have the judge read figures computed by arithmetic rather than prose, wherever a criterion can be settled that way.

Nobody can rerun a model's run, and the field has stopped pretending otherwise. Three quarters of the 75 studies in the Kempny review report no model parameters at all. Chew and colleagues put the reason plainly, that generative models are stochastic and the same output cannot be reproduced later, and propose publishing prompts, model details, texts, codes and the model's own reasons instead of promising reproducibility. When the computation cannot be rerun, the auditable object is the plan sealed in advance plus the full record of what was sent and what came back.

Two gaps to state. This search found nothing on LLMs used for process tracing, contribution analysis or theory-of-change testing specifically, so the application is unexplored rather than established. And every figure above is bound to a model vintage between GPT-3 and Llama 3, which is the same argument Dunivin's own GPT-3.5 to GPT-4 gap makes: these numbers describe the past, not the models in use now.