AI in Evaluation
11 pages in this theme.
An evaluator niche in AI
Can evaluators find a niche in auditing whether AI applications are trustworthy, culture aware, valid and transparent? What would need to change in AI systems to make them suitable for a transformatio…
Asking your own AI about your causal maps
This page is for people who code their data in Causal Map and also have Claude, ChatGPT or another AI chat app open while they write. It explains what changes when the chat app can read your causal ma…
ChatGPT - causal, of course
! We can thank Judea Pearl for promoting the insight that if you want to thrive in this world, you have to understand causality natively. We humans make causal connections from an early age. We wouldn…
ChatGPT is changing how we do evaluation. The view from Causal Map.
! Causal mapping – the process of identifying and synthesising causal claims within documents – is about to become much more accessible to evaluators . At Causal Map Ltd, we use causal mapping to solv…
How hard is evaluation actually
! 🏭 When machines replaced much manual labour, white collar workers thought "I'm ok, my job is much harder to mechanise". 🖥 And then when computers came for clerical jobs, university educated white co…
Pilots, not passengers – reconstructible reporting in evaluation
For evaluators, commissioners and anyone else working out what evaluation analysis should look like now that AI can do much of it. In our experience, a frontier model such as Claude Opus 5.5, given th…
The car won't start
There is a steady stream of advice at the moment telling us to stop saying that an AI knows, wants, thinks, decides or tries. One recent example sets out how to talk about AI without adding to the hyp…
The point of us – when AI can do the desk analysis, the learning between people is the output
For evaluators, commissioners and evaluation managers who are asking what an evaluation is for, now that AI can do much of the analysis. Work in progress. A position paper. Most of it argues from our…
What is the Point of Us. A Sci-Fi Story for Researchers
I was talking to my friend Alberta yesterday about the open letter signed by prominent proponents of thematic analysis, rejecting the use of generative AI tools in qualitative social research. We were…
Yes, it's ok to say that an AI understands what you say
Me: "ah ChatGPT misunderstood when I said 'United' I meant Sheffield United not Manchester United". Some pedant: "you shouldn't say 'understand': an AI is not even a robot. It is just a large language…
Yes, there are AI-shaped holes in organisations.
Matthew Clifford says: “There are no AI shaped holes lying around”. That is how he reconciles "the facts that (a) AI is already powerful and (b) it’s having relatively little impact so far Making AI w…