⛶
reconstructability-v2

Reconstructability, and the well defined question#

DRAFT version 2. Supersedes reconstructability.md and reconstructability-nachvollziehbar-section.md, both of which can be deleted. Argument structure settled; prose needs Steve's voice over it, and the citations need checking.

The long struggle#

For the better part of two decades I have been trying to say something I could never quite get into words. In Causal Map we called it codeability, or the well defined question. The idea was that a question is well defined if it has been broken into pieces small enough that two competent people, working separately, could be expected to reach roughly the same answers.

The trouble is that this is a definition that eats itself. What counts as small enough? What counts as roughly? Every attempt to specify it drags you into the philosophy of meaning, and you end up arguing about the nature of questions rather than getting on with the evaluation.

Consider "summarise this document". It looks innocuous. It is not. Ask it on Tuesday and again on Thursday, of two different models, or of two different people, and you get answers that differ in ways you cannot predict. Nothing about the phrasing tells you that. There is no mark of vagueness on its face.

The sideways move#

Here is another way to ask the same question, which turns out to be much easier to answer.

Take the workflow you wrote for your evaluation. The sources, what gets marked and against what criteria, how things are counted, what standard turns a count into a verdict. Hand that document, and only that document, to a competent stranger who knows nothing else about your project. No conversation with you, no shared context, no paraphernalia.

Can they carry it out, and land somewhere recognisably the same as you did?

That is reconstructability. And the striking thing is that it never requires you to define what makes a question well defined. It tests for it. "Summarise this document" fails, not because we have argued it is vague, but because two reconstructions visibly diverge.

Vagueness stops being a philosophical property and becomes an observable one.

Why this is not reproducibility#

The word reproducibility will make a large part of the evaluation profession stop reading, and with good reason. It carries the demand that you run the analysis twice and get the same numbers, which qualitative work cannot do, should not pretend to do, and is not diminished by failing to do.

Reconstructability asks for something different.

It is the workflow that is reproduced, not the findings. The results of a second run will differ: different emphases, borderline cases called the other way, a different model on a different day. That is expected and it is fine. What must survive the journey is the method. The second party must be doing recognisably the same thing to the same material for the same stated reasons.

The claim is not that you would get the same answer. It is the weaker and more useful one: that there was an answerable question here, specified tightly enough for someone else to take it up.

The soft version, and why German has the better word#

There are two strengths of this test, and English has a word for neither.

German has one for the softer of them: nachvollziehbar. Literally, something like able-to-be-followed-through. A piece of reasoning is nachvollziehbar if another person can retrace the path you took and see how you got from where you started to where you ended up. Crucially, it does not require them to agree with you. Your conclusion can be one they would never have reached, and the reasoning can still be entirely nachvollziehbar. What is being tested is whether the path is visible, not whether it was the right path.

English translations all lose something. Comprehensible is too weak and says only that the sentences parse. Traceable sounds like logistics. Intelligible is about clarity rather than about following. The nearest thing in our own literature is Lincoln and Guba's audit trail, which is a procedure rather than a property, and Sandelowski's auditable, which is closer but has acquired an accounting smell.

So: the soft version of reconstructability is nachvollziehbar. A competent human reads your workflow and your report, and can follow how one produced the other. This is a real and valuable standard, and most evaluation reports fail it.

The hard version is the one this piece is mostly about. Give the workflow, and only the workflow, to a naive model. Not follow it: run it. Land somewhere recognisably the same.

These are not two different ideas. They are the same property, tested by two different auditors, and the difference between them is one of degree rather than of kind. A workflow that a model can execute is certainly one a human can follow. A workflow a human can follow may still be far too loose for a model to run, because the human silently repairs the gaps as they read, and does not notice doing it.

That last point is the useful one. The value of the hard test is not that machine execution is somehow more authoritative than human judgement. It is that the machine does not do you the favour of filling in what you forgot to say.

Two ways to fail, one test#

A failed reconstruction tells you which of two things went wrong, and both are worth knowing.

The specification was too loose. Steps that assumed knowledge only you had, criteria that sounded definite but were not, a threshold never actually stated. Common, and usually fixable in an afternoon once exposed.

Or a step was inherently too stochastic to be carried out consistently by anyone. Not badly written, but asking for a judgement so unconstrained that no two readings could converge. That is a finding about the analysis itself, and no amount of rewriting will fix it. The step needs replacing.

Both failures are invisible while the work stays in your own head, where every ambiguity is silently resolved by context you are not aware of having.

What it does not touch, and this matters#

Reconstructability applies to the workflow once it has been decided and written down. It says nothing whatsoever about how you arrived at it, and it is not an argument that the arriving does not matter.

At one extreme the workflow is handed to you: agreed with the commissioner in advance, or simply required. At the other it is the product of ten years in a loft in Heidelberg, thinking about the philosophy of causation, reading, arguing, changing your mind, noticing your own position in the work and what that does to what you see.

Some of that reasoning can be written down reconstructably. A great deal of it cannot, and the parts that cannot may be the most valuable of all. The reflexive turn that made you realise you were asking the wrong question does not decompose into steps, and nothing here suggests that it should.

The test begins after that. Once you have decided how you are going to read this evidence, write it down in a way that means something to someone who was not in the loft.

Why now#

The test has been available in principle for forty years. Lincoln and Guba described the audit trail in 1985, and Sandelowski wrote of findings being auditable when another researcher can clearly follow the decision trail used by the investigator. The idea is not new.

What was missing was the auditor. A naive, competent, patient reader with no other knowledge of your project, willing to follow your workflow to the letter and report where it broke, was until recently expensive and scarce. Nobody was going to spend a fortnight of a colleague's time proving their own method was sound.

Now that reader costs pennies and takes minutes. The old standard has become operational, and that changes what we can reasonably ask of each other.

Notes for writing, to be deleted before publication#