⛶
blog-draft-two-tests

Drafted 11 September 2026, from conversation. Working draft, not finished copy. Companion to the reconstructability post — second in a pair, and it assumes the first. The argument structure is the valuable part; the prose needs Steve's voice over it.

A second test#

The previous piece argued for reconstructability: hand your written workflow to a competent stranger, and see whether they can carry it out and land somewhere recognisably the same. It tests whether a question was well defined, without ever requiring you to say what well defined means.

But it has an obvious hole, and it is worth saying out loud rather than waiting to be caught.

A workflow can be perfectly reconstructable and still useless. "Count the number of times the word empowerment appears, and if it is more than ten, conclude the programme worked." A stranger could follow that to the letter. Two strangers would agree. It is specified beautifully. It also cannot tell a good programme from a bad one.

Reconstructability is a test of the specification. It is not a test of whether the thing specified can see anything.

The discrimination test#

So, the second test. Generate synthetic texts — cases you have constructed, so you know the answer because you built it in — where some should come out one way and some the other. Run your workflow over them. Does it tell them apart?

If your workflow is supposed to detect whether a report describes a programme influencing a decision, then write some reports where it plainly does and some where it plainly does not, and some near-misses where a person mentions the programme and the decision in the same breath without any causal claim at all. If the workflow marks all of them the same way, it is not measuring what you think it is measuring, however cleanly it is written down.

There are many ways of constructing such texts, and the construction is where the craft lives. But the principle is simple: a measure that cannot separate cases that differ is not a measure.

Where these sit, for those who care about lineage#

Neither test is new, and it helps to say what they are relatives of.

Reconstructability belongs with transparency and auditability. Lincoln and Guba called it dependability and confirmability in 1985 and described the audit trail; Sandelowski wrote of findings being auditable when another researcher can follow the decision trail. In political science the same impulse produced DA-RT and Annotation for Transparent Inquiry. The nearest thing in computing is the ACM's Results Replicated badge, which is awarded when someone obtains a paper's main results independently, without the author's own code and data — reimplementation from the description alone.

The discrimination test belongs with construct validity, in the sense Cronbach and Meehl gave it in 1955: work out what your measure ought to do if it is measuring what you claim, then check that it does. More specifically it is a cousin of discriminant validity, Campbell and Fiske, 1959 — though not the same thing, and the difference is worth respecting. Campbell and Fiske were talking about two measures failing to correlate across a sample of units. We are talking about one workflow separating two constructed texts. The intuition is shared; the machinery is not. In form it is closer to a known-groups design, or to sensitivity and specificity.

On the computational side the direct ancestor is behavioural testing. CheckList, Ribeiro and colleagues in 2020, borrowed the idea from software engineering: build minimal pairs that differ only in the thing the system is supposed to notice, and see whether it notices. More recent work has language models generate the test cases. The difference is that CheckList tests a model. We would be testing a workflow — a longer object, with more places to go wrong, and one that a practitioner rather than an engineer has written.

What the two tests do not cover#

This is the part that matters, and the reason for writing the piece.

If you are going to claim two tests, you should be able to say straight away what they are not good for. Otherwise the first person who finds a gap gets to frame it as a flaw you were hiding.

They say nothing about whether yours was the right workflow to write. Discrimination shows the workflow can separate cases given your concept. Reconstructability shows the procedure is tight enough to follow. Both hold the workflow fixed. Neither asks what a reasonable colleague would have got with a different threshold, a different concept boundary, a different reading of what the commissioner meant.

In statistics you would answer this with a robustness check: vary the specification, see whether the conclusion survives. That works because the specifications lie on a common scale. Here they do not. There is no metric over operationalisations of "empowerment", so the spread across several of them tells you only that people differ — which we knew.

So we are leaving it out, deliberately, and this is the honest version of why: the duty is to make the choice visible enough to be disputed, not to prove it was the best available. A reader who thinks you drew the boundary in the wrong place can see exactly where you drew it, and say so. That is a real service. It is not the same as robustness and should not be sold as it.

They say little about calibration. Whether a workflow's stated confidence tracks reality is a further question, and a live one given the documented tendency of language models to overstate likelihoods. Treat it as a refinement of discrimination rather than a third pillar, but do not pretend a binary separation is the whole story.

They do not check the concept against the world. Synthetic texts test the workflow against the concept as you encoded it. If your encoding and your commissioner's understanding of the word have quietly diverged, the workflow passes both tests and measures the wrong thing beautifully. Whether a formulated question spans the reasonable readings of it is a proper question, and one for later.

The short version#

Both tests are about the workflow as written. One shows it can see; the other shows it can be followed. Neither says it was the right workflow to write — and writing it down is what makes that arguable.

Notes for writing#