Journal › Checking the advice
How do I know if the AI advice I'm reading is any good?
The short answer
Check whether anyone tested it, and whether it held up when somebody repeated it. Most advice about AI gets repeated because it sounds right, not because it was ever checked. I had the most-quoted techniques in my own field traced back to the original research this month, and three of them fell over.
I was about to change how I write this diary, following advice that gets repeated everywhere. Before I did, I had it checked against the original research. Three of the techniques did not survive — including the one behind every cliffhanger you have ever been shown. So I did not use them.
What I actually did
The first was the “open loop” — the idea that an unfinished story sticks in your mind and pulls you back. It is the single most-cited justification for cliffhangers in marketing, and a 2025 meta-analysis found no memory advantage for unfinished tasks at all. The second was nudging, the small changes to how a choice is presented, which reported a solid effect until a second team corrected for the fact that studies finding an effect get published far more often than studies finding none. The third, “if-then” planning, survived — but at a fraction of the size it is sold at. The people repeating these are not lying. They are repeating what they read, which was repeating what somebody else read. That is worth knowing whoever is selling you something, and it is why I would rather tell you what I checked than what I concluded.
What you can check
- The open loop: a 2025 meta-analysis pulled together 59 published studies. Across the 37 testing recall, once Zeigarnik’s own 1927 data is excluded, interrupted tasks are recalled no better than finished ones — a ratio of 0.99. Ghibellini & Meier, Humanities and Social Sciences Communications, doi.org/10.1057/s41599-025-05000-w
- Nudging: a large analysis reported 0.45 and noted “a moderate publication bias toward positive results”. Corrected for that bias, the effect fell from 0.43 to 0.04, and for nudges that work purely by giving people information it fell to 0.00. Their own caveat travels with it: individual nudges may still work, it is the average that does not survive. Mertens et al., PNAS 119(1); Maier et al., PNAS 119(31), doi.org/10.1073/pnas.2200300119
- If-then planning: the 2006 review put it at 0.65. The same authors later re-analysed 642 tests correcting for publication bias and reported 0.15. It still works. It is not the lever it is sold as. Sheeran, Listrom & Gollwitzer, European Review of Social Psychology 36(1), doi.org/10.1080/10463283.2024.2334563
- If you want one question to ask of any technique you are sold, ask who tested it and whether anyone repeated it. A finding nobody has repeated is a hypothesis.