Rigour and Evidence
18 pages in this theme.
Assessing quality or robustness of evidence for a causal link based on a bundle of coterminal causal claims
This post gives more details on one of the key moments for Quality Assurance in causal mapping, the bundle assessment, which is Step 6 of the 901 A workflow for causal coding with and without AI ((ai…
Checking an AI answer against a published study – MECC train the trainer
For evaluation commissioners and practitioners wondering whether an AI assisted analysis of interviews can be trusted, and how they would know. Part of 000 Pilots, not passengers reconstructible repor…
Checking the claims in an AI report cannot find its omissions
For evaluators and applied social researchers who use AI in qualitative analysis, or use or assess work that does. Summary A frontier AI can turn a terms of reference and a pile of transcripts into a…
Epistemic logic does not help us with reasoning about causal maps
(An example of kind of qualitative causal logic, with a focus on groups: @castellaniCasebasedSystemsMapping]) From [@powellCausalMappingEvaluators2024] Seen as models of the world, causal maps, like s…
Just add rigour Three do’s and don’ts
!500 AI in qualitative social science/img/map 050 just add rigour three do s and don ts add rigour.png Three do’s and don’ts when using AI for text analysis. A lot of evaluation work is a kind of text…
Limitations on causal mapping in evaluation - data quality
From [@powellCausalMappingEvaluators2024] Causal mapping has some limitations. First, the credibility of the causal arguments which can be derived from a map is limited by the credibility of the origi…
Link metadata – quality of evidence
A 410 Adding and using custom columns for your links ((howto custom columns))|custom column can record something about the quality of the evidence behind each link, for example a confidence score, or…
Our approach clearly distinguishes evidence from facts and does not automatically warrant causal inferences
From Better Evaluation. Causal mapping distinguishes carefully between evidence for a causal link and the causal link itself. It does not provide any specific way to make causal inferences from one to…
Principles for reconstructible analysis in the age of AI
For evaluators who use AI to analyse interviews, reports and other text, and for the people who commission their work. These are the principles we work to when AI does some or all of an analysis of te…
Qualitative impact evaluation is less interested in the strength of effects
! Weird image of people counting beans generated by Canva's AI Soft versus hard impact evaluation approaches? Quant versus Qual? Is there an essential difference? Summary: quant and qual impact evalua…
Quality assurance at each step of the causal coding workflow
Summary: Quality assurance at each step In the qualitative space, evaluators have many tools and approaches for reaching robust and rigorous conclusions about causal influences on an outcome of intere…
Robustness – How robust is the evidence for that X influences Y ?
How robust is the network of evidence for the influence of one or more "driver" factors on another set of "outcome" factors? From [@powellCausalMappingEvaluators2024] We prefer to talk about possible…
Strong evidence for a link is not evidence of a strong link
Never confuse the two.
The elephant in the room – causal inference
! Responding&dashCommentUrn=urn%3Ali%3Afsd comment%3A(7008516217275650050%2Curn%3Ali%3AugcPost%3A7008450782928732160)&dashReplyUrn=urn%3Ali%3Afsd comment%3A(7008865880012988416%2Curn%3Ali%3AugcPost%3A…
The jury test
Work in progress. Which steps an analysis may be built from if somebody else is to rebuild and check it, and how far to break it down. Part of 000 Pilots, not passengers reconstructible reporting in e…
The transitivity trap
From [@powellCausalMappingEvaluators2024] Transitivity is perhaps the single most important challenge for causal mapping. Consider the following example. If source P [pig farmer] states ‘I received ca…
Ways to causal inference
There are different ways to set up a table like this. Some of these are more like methods, some are more like frameworks. Some overlap. Most do not exist only or even primarily to conduct causal infer…
We can reason about causal maps using a logic of evidence
From [@powellCausalMappingEvaluators2024] Evaluators can break the Janus dilemma and make the best use of causal maps in evaluation by considering causal maps not primarily as models of either beliefs…