Baloney detection kit
Last reviewed:
Carl Sagan set out a kit for testing a claim in The Demon-Haunted World (1995), chapter 12, “The Fine Art of Baloney Detection.” The card in this library is a summary of that chapter, not a quotation of one sentence. This page splits that summary into the tools, gives each a short illustration that is not a scene from the book, and then prints the three practice questions already on the card.
Carl Sagan's toolkit for testing a claim, from The Demon-Haunted World (1995), chapter 12, 'The Fine Art of Baloney Detection'. In summary, his tools are: wherever possible, independent confirmation of the facts; substantive debate among people who know the evidence; little weight on arguments from authority, because authorities have been wrong and science has experts rather than authorities; more than one hypothesis, and tests that could rule each out; not getting attached to an idea just because it is yours; quantifying when the thing can be quantified; every link in a chain of argument has to work, not most of them; Occam's razor when two hypotheses fit equally well; and the question whether the hypothesis could, in principle, be falsified. This card is a summary of that chapter, not a single sentence of his.
1. Independent confirmation
Wherever possible, look for an independent check of the facts.
Illustration, not a case from the book: a cure is described only by the person selling it. The kit asks for someone who does not profit from the claim to look at the same facts.
2. Substantive debate
Encourage debate among people who know the evidence, not just people who share a mood.
Illustration: a group chat agrees quickly. The kit asks whether anyone in the chat has actually handled the evidence, and whether disagreement is allowed to finish a sentence.
3. Little weight on authority
Authorities have been wrong. Science has experts rather than authorities. A name is not the last word.
Illustration: a famous doctor is quoted outside the work they do. The kit does not say to ignore people who know a field. It says not to stop at the name.
4. More than one hypothesis
Spin more than one hypothesis, and look for tests that could rule each of them out.
Illustration: the first story that fits is treated as the only story. The kit asks what else fits the same facts, and what would rule each story out.
5. Don’t fall in love with your idea
Do not keep a hypothesis because it is yours.
Illustration: two explanations fit, and you prefer the one you thought of on the way there. Attachment is not a reason. This is the second practice question on the card.
6. Quantify
When the thing can be quantified, quantify it.
Illustration: a story says “everyone” or “always” and nobody asks for the count the story is standing in for. If it cannot be counted, say so, and do not pretend a feeling was a count.
7. Every link has to work
In a chain of argument, every link has to work, not most of them.
Illustration: four steps are offered and one of them is only a hope. A longer chain is not a stronger chain when a link is missing.
8. Occam’s razor
When two hypotheses fit equally well, prefer the one that asks you to assume less.
Illustration: both stories fit the messy facts. The kit does not say the simpler story is true. It says not to add extra machinery you do not need.
9. Falsifiability
Ask whether the hypothesis could, even in principle, be shown false.
Illustration: a friend says the claim cannot be tested, and that this is what makes it deep. The third practice question on the card is this tool. If nothing could count against it, you do not yet have a finding.
Practice questions
These three are the questions on the card. The button opens them in the quiz.
A cure is backed only by the person selling it. Which move from Sagan's kit is the one to reach for first?
Chapter 12 puts independent confirmation first. The seller is one source, and an interested one. Authority and affection for your own hope are later tools on the same list, and they cut the other way.
Two explanations fit the same messy facts. You prefer the one you thought of on the bus. What does the kit say?
Sagan asks for multiple hypotheses and warns against attachment to your own. A longer chain with a broken link is worse, not better. Two live ideas are a reason to test, not a reason to stop.
A friend says the claim cannot be tested, and that this depth is what makes it impressive. Which tool answers?
Falsifiability is one of the tools in chapter 12. A claim arranged so that nothing counts against it has failed that tool. Applause and authority do not replace it.
FAQ
Where does the kit come from?
Carl Sagan, The Demon-Haunted World (1995), chapter 12, “The Fine Art of Baloney Detection.” The card in this library is a summary of that chapter, not a single sentence of his.
Is this page quoting Sagan line by line?
No. The tools are paraphrased from the summary already on the Baloney detection kit card. The illustrations are not scenes from the book. His filed sentences, including the dragon in the garage and the balance of skepticism and wonder, are on his philosopher page.
What should you read?
His page says to start with The Demon-Haunted World, chapter 12, then the opening reflection in Pale Blue Dot. The kit is the method. The photograph is the scale.