5. What counts as evidence? | Quomodo Scimus : IBTOK

The Core Theme - Knowledge and the Knower

Lesson 5 - What counts as evidence? (Methods and Tools 1)

Intelligent people can be systematically wrong for extended periods - and understanding why matters as much as understanding how to get things right.

THE PROVOCATION

The Mask Reversal

The experts were wrong in a way that turns out to be extremely interesting.

Dr. Anthony Fauci, March 2020
60 Minutes

In March 2020, Dr. Anthony Fauci, Director of the NIAID, became the scientific face of the U.S. government's COVID-19 response. As a lead member of the White House Coronavirus Task Force, he used daily televised briefings to explain complex viral dynamics to a panicked public. In this clip from March 2020 he advises about the use of face masks in public spaces.

You just watched it. In March 2020, the most senior infectious disease official in the United States told the world there was no reason to wear a mask. The WHO said the same. So did the UK government, the French government, the Swiss government, and virtually every other scientific establishment in the world. It was the consensus of the best-credentialed experts in the world. Six weeks later, they reversed that decision - masks were now essential. The guidance that had been given to billions of people was wrong.

This lesson begins with a specific question: how experts who were doing their job correctly, making rational inferences from incomplete evidence, arrived at a conclusion that turned out to be wrong. They were doing what all knowers do. They were being rational, and they were still wrong.

Understanding how this is possible - how intelligent, highly qualified, well-intentioned experts can be systematically wrong, and why correcting them takes so long - is the central question of western epistemology. It is also one of the most practically important questions you will encounter in your lifetime. The tools for thinking about it were developed over three centuries, and one of the clearest illustrations comes from history and a street map drawn by a London anaesthesiologist in 1854. That map is where this lesson begins, because before we can ask how knowledge is justified, we need to see what justified knowledge actually looks like when it works, and what tools made it possible.

Before we begin: The language of arguments

A Toolkit for Evaluating Claims

Before examining how scientific reasoning works, it helps to have precise vocabulary for what counts as a good argument. These three terms function as analytical tools - instruments for examining the structure of claims rather than just their content. They will appear throughout the rest of the course, particularly in Social Sciences, where the quality of reasoning behind a claim is always the first question to ask.

When scientists told the public in early 2020 that masks were not necessary, the argument was inductive. Inductive reasoning draws conclusions from evidence: the conclusion is probable given what is known, but never logically guaranteed. The mask argument ran roughly as follows. Available evidence suggested the virus spread mainly through large respiratory droplets. Masks at the population level were judged to offer limited additional protection. The conclusion followed reasonably from those premises. But inductive conclusions are only as strong as the evidence behind them. When evidence accumulated for airborne transmission through smaller aerosol particles, the premises changed, and the guidance changed with them.

Deductive reasoning works by a different standard. In a valid deductive argument, if the premises are true, the conclusion must be true - there is no room for doubt or exception. The mask argument could not have been deductive, because it rested on incomplete empirical knowledge, and empirical knowledge about a new virus is always incomplete.

Understanding the difference matters because the two types of argument are assessed by different criteria. A deductive argument is valid or invalid, sound or unsound - these are the terms from the infographic above. An inductive argument is assessed by how well-supported the evidence is, and how carefully the conclusion follows from it. Science depends on induction; mathematics depends on deduction.

Use these terms as tools. When you encounter a knowledge claim, ask which kind of argument is being made, whether the premises are well-supported, and whether the conclusion has been drawn carefully.

Four big ideas about how we claim to know what we know.

Big idea 1 - Evidence does not speak for itself

Knowing which explanation is right requires something beyond the facts alone, and that something is harder to find than it looks.

"He reviewed Farr's list again, looking this time for telltale absences."
Steven Johnson, The Ghost Map (2006)

This lesson should really be about the scientific method, but because I'm a history teacher it isn't; at least not directly. Instead, it combines two of my favourite things Victorian Britain and the writer and Steven Johnson (Mr. NotebookLM). In fact, I do address the background of this in a lesson in DP1 but you probably missed it. But this time its unavoidable because I'm going to integrate this story into each of the four big ideas that make up this section. But first the story.

A guided tour of the Ghost Map
Steven Johnson - TED

Steven Johnson is a prominent media theorist and author whose work explores the history of innovation. In his acclaimed book The Ghost Map, he chronicles the 1854 London cholera outbreak, and today Johnson applies these principles of "networked thought" as the Editorial Director of NotebookLM at Google. He helped develop the AI tool to serve as a collaborative research assistant, empowering users to synthesise complex information and generate new insights, and allowed me to make many podcasts and infographics on this website.

It was also completely wrong. The cause was contaminated water from a single pump on Broad Street. A methodical anaesthesiologist named John Snow proved it using a map - this little map below - it is one of the most important maps ever made.

In the late summer of 1854, a disease killed 616 people in ten days in a single neighbourhood of London. Nobody knew what caused it. The city's leading medical authorities were certain of the answer: miasma, bad air rising from the sewers and the Thames. This was not a fringe theory. Edwin Chadwick, who had done more than any person alive to improve London's public health infrastructure, was a passionate miasmatist. Florence Nightingale was a miasmatist. The General Board of Health was staffed by miasmatists. The consensus was overwhelming.

It was also completely wrong. The cause was contaminated water from a single pump on Broad Street. A methodical anaesthesiologist named John Snow proved it using a map, this little map on the right. You should click on it and enlarge it. It is one of the most important maps ever made.

Snow plotted every death on a street map of the neighbourhood. The pattern was unmistakeable. The deaths clustered around the pump. But the map alone did not prove his theory - the miasmatists could argue that the same area had the worst-smelling air in the district, and that the bad air, not the water, explained the cluster. The same data can, in principle, support more than one account of what caused it.

What made Snow's investigation decisive was what he did next. Rather than look for further evidence that confirmed he thesis, he went looking for cases that should have appeared on the map but did not. He tried to disprove his theory. Across the street from the pump stood a workhouse with 535 residents. If the water or bad smells was killing people, the workhouse should have accounted for dozens of deaths. There were two. Snow discovered that the workhouse had its own well. Nearby, a brewery employed hundreds of men who worked long hours in the heat. None had died. The brewery provided workers with a daily ration of malt liquor. They never drank from the pump. These absences were as important as the deaths on the map. Snow approached the outbreak with a hypothesis: the cause was contaminated water, and the Broad Street pump was the source. This is what hypothesis-driven inquiry means. Evidence is not simply found. It is produced through a process of directed search, in which the hypothesis determines what counts as a relevant observation. What comes back either supports it or forces it to change.

In early 2020, the hypothesis guiding mask guidance was that respiratory viruses spread primarily through large droplets that fall quickly to the ground and not through fine aerosols that hang in the air. The evidence available was consistent with this picture. What nobody was systematically looking for were the anomalous cases, the superspreader events in poorly ventilated indoor spaces where no one had been close enough for droplet transmission to explain the spread. More on this later.

Every claim we make about the world that goes beyond what we have directly observed is an inductive inference. And induction can never be fully justified.

Big idea 2 - Induction: the engine of knowledge, and its problem

"Hume, I felt, was perfectly right in pointing out that induction cannot be logically justified. He held that there can be no valid logical arguments allowing us to establish 'that those instances, of which we have had no experience, resemble those, of which we have had experience.'" — Karl Popper, Conjectures and Refutations (1963)

Again we have encountered the content of this lesson in a history lesson, this time in the far distant past of 11e. But before we go any further, here is a splendid NotebookLM generated infographic to remind you of the difference between inductive and deductive reasoning.

The two great forms of reasoning are deduction and induction. Deduction moves from premises to a conclusion that necessarily follows: if all humans are mortal and Socrates is human, then Socrates is mortal. The conclusion cannot be false if the premises are true. But deduction generates no new knowledge - you can only extract what was already implicit in the premises.

Induction moves in the opposite direction, from particular observations to general conclusions. Snow examined the geographic distribution of cases across two outbreaks and concluded that contaminated water was the cause. Every contact tracer mapping a transmission chain in 2020 was reasoning the same way: these cases share this exposure, so that exposure is probably the source. Every R number and every vaccine efficacy figure was an inductive inference from observed data to a general claim about how the virus behaves. Induction is the engine of empirical knowledge. David Hume, writing in 1748, showed it is also incapable of logical justification.

Hume's argument is simple and devastating. The inference from past experience to future expectation has no logical foundation - the conclusion does not follow necessarily from the premises. And if you try to justify induction by saying that it has worked reliably in the past, you are using induction to justify induction, which is circular. Hume's conclusion: the belief that the future will resemble the past - the foundation of all empirical knowledge - cannot be rationally secured.

Hume's problem sits, unresolved, underneath every prediction made during the pandemic. The models that projected mortality, the trials that established vaccine efficacy: all of it rested on inductive inference, on a foundation Hume showed we cannot fully secure. We act as though induction justifies our beliefs because we cannot do without it.

Big idea 3 - Popper's response: the logic of falsification.

A scientific theory can be refuted but never finally proved. That asymmetry is what gives science its distinctive power.

"It is easy to obtain confirmations, or verifications, for nearly every theory - if we look for confirmations. Confirmations should count only if they are the result of risky predictions... an event which would have refuted the theory. Every 'good' scientific theory is a prohibition: it forbids certain things to happen. The more a theory forbids, the better it is."
Karl Popper, Conjectures and Refutations (1963)

Popper accepted Hume's problem and reframed what science is doing. The proper aim of science is attempted refutation: exposing theories to the harshest possible tests rather than accumulating confirmations. The relevant question to ask of any empirical claim is: what evidence would falsify it? Any theory can be made to look confirmed if you search only for supporting instances. What distinguishes a scientific theory is that it specifies in advance what would count as evidence against it. A theory that cannot in principle be falsified is not a scientific theory at all.

No number of confirming instances can prove a universal claim. Seeing a million white swans supports the claim that "all swans are white." This is an inductive generalisation. For centuries in Europe, all known swans were white. Then in 1697, Dutch explorers led by Willem de Vlamingh arrived in Western Australia and encountered black swans. A single observation overturned a long-standing "truth." A single black swan refutes the claim with logical certainty. This is why Popper argued that the proper structure of scientific reasoning is exposing theories to the harshest possible tests, not accumulating confirmations.

Falsifiability is a diagnostic tool as well as a logical principle. It allows you to ask of any knowledge claim: what would have to be true for this to be wrong? If nothing could count as evidence against the claim, it is not science. Freudian psychoanalysis, Popper argued, was unfalsifiable in this sense because any behaviour could be reinterpreted to fit the theory, and the theory specified in advance no evidence that would refute it. That did not make it false or useless - it simply made it a different kind of claim from a scientific one. The tool of falsifiability draws a boundary between inquiry that can be corrected by evidence and inquiry that cannot.

Karl Popper's Falsification
BBC Radio 4

Popper was born in Vienna and spent most of his academic career at the London School of Economics. His philosophy of science, developed in The Logic of Scientific Discovery (1934) and Conjectures and Refutations (1963), proposed falsifiability as the criterion distinguishing scientific claims from non-scientific ones. His arguments against Marxism and Freudianism as unfalsifiable - theories that could accommodate any evidence by reinterpretation. ( The Open Society and Its Enemies and The Poverty of Historicism)

In a way Snow was a Popperian before Popper. He went looking for the cases that should have destroyed his theory. The workhouse with 535 residents and only two deaths was exactly such a case. Snow investigated the anomaly rather than ignoring it, and the explanation - a private well that provided a clean source of water - confirmed the theory precisely by explaining away the apparent counterexample.

The droplet/aerosol debate in 2020 had a Popperian structure too, but the falsifying tests were slow to arrive. The superspreader event in a Skagit Valley choir rehearsal in March 2020 - where 52 of 61 singers were infected after a two-and-a-half hour rehearsal, despite the absence of anyone visibly ill - was exactly the kind of unusual result that should have triggered urgent investigation of the droplet hypothesis.

It took months for the aerosol evidence to accumulate to the point where official guidance shifted. The problem was that the scientific community was trapped in a questionable paradigm - and the community of knowers were a very bad influence on each other. Big Idea 4 explains why.

Popper's insight looks even sharper in the age of artificial intelligence. Large language models - the systems behind ChatGPT, Claude, Gemini, and every other AI assistant you use - are trained on human approval. They learn by producing outputs that human evaluators rate as helpful, plausible, and satisfying. This means they are structurally optimised to confirm, to find reasons a claim sounds right, to produce answers that feel convincing, to agree rather than to challenge. This is precisely the habit Popper identified as epistemologically dangerous. Confirmations are easy and cheap. What distinguishes rigorous reasoning is the willingness to ask: what would show this to be wrong? An AI system trained to please human evaluators has been built to answer a different question entirely.

Big idea 4 - The social life of knowledge

Snow was right and the establishment was wrong. It took them a decade to accept it. Understanding why is as important as understanding Snow's method.

"How could so many intelligent people be so grievously wrong for such an extended period of time? How could they ignore so much overwhelming evidence that contradicted their most basic theories? These questions, too, deserve their own discipline - the sociology of error."
Steven Johnson, The Ghost Map (2006)

The philosopher Thomas Kuhn, writing a century after Snow, gave the most systematic account of why Snow faced resistance for his theory. Scientific knowledge is produced by communities of knowers who share a paradigm - a set of assumptions, methods, exemplary problems and solutions, and a picture of what the world is like - that structures what they can observe, what questions they can ask, and what counts as an acceptable answer.

No scientist starts from first principles each time they enter a laboratory. Science requires shared assumptions about what counts as a valid method, a reliable instrument, and a meaningful result. Without them, inquiry cannot begin. Kuhn called this framework a paradigm: the background commitments that make normal science possible. The paradigm enables inquiry precisely because it is not itself under constant investigation. It tells you what to look for and what would count as an answer.

But what makes it productive also makes it resistant to challenge. When results do not fit the framework, they are set aside, reinterpreted, or attributed to experimental error. A paradigm holds until the accumulation of anomalies becomes too great to contain and a new framework emerges that can accommodate them better. Kuhn's deeper point is that this process is not purely logical. Paradigm shifts involve generational change in scientific communities and contests over institutional authority: who controls the journals and the funding, and who has the power to define what legitimate research looks like. You will remember the Galileo example: his evidence was measured against a framework the Church had no reason to question, and found to be in violation of it. That is what Kuhn means by normal science.

The mask guidance reversal was also, in part, a communication problem with an epistemological dimension. The early advice against masks was partly strategic and designed to protect the supply for healthcare workers. When the science changed, distinguishing what had been a justified scientific claim from what had been a strategic communication was nearly impossible from the outside. Also by this point there were enough masks to go around. This is what happens when the social and epistemic functions of knowledge-making institutions become entangled.

Paradigm Shift
Social Science Explainer

Thomas Kuhn (1922–1996) was an American historian and philosopher of science best known for The Structure of Scientific Revolutions. He argued that science does not progress steadily but through disruptive "paradigm shifts," where dominant frameworks are replaced after crises and anomalies accumulate. A paradigm shapes what counts as knowledge, methods, and truth; when it collapses, a new worldview emerges. Kuhn's work challenged the idea of purely objective, cumulative scientific progress.

Bringing it together

The method and the tools are inseparable.

Go back to the clip you watched at the start. The Fauci interview was a claim made in good faith, on the basis of available evidence, within a paradigm that structured what counted as relevant and what did not. Hume tells us why more data alone was not sufficient: every inference from that data to a general claim about transmission was inductive, and induction cannot guarantee its conclusions. Popper tells us what should have happened faster: the anomalous superspreader events should have been treated as falsifying tests, not as noise. Kuhn tells us why they weren't: the droplet paradigm had become an infrastructure, embedded in labs, protocols, and funding decisions, and infrastructures resist revision.

Snow's story adds the tools dimension to the scientific method. His map transformed the data: the spatial visualisation made a pattern visible that the mortality tables could not show. This is a great TOK observation and also worth remembering in history when you're asked about the value of a graph. The absence Snow looked for at the workhouse was found because his hypothesis and map told him where to look. The pump handle was removed because he had a falsifiable prediction. This is what methods and tools in combination look like: a method (falsificationist inquiry) applied through specific instruments (the map, the anomaly-search, the controlled intervention) in a social context (the Board of Health, the competing paradigm, the eventual institutional acceptance).

Snow died in 1858, four years after the Broad Street outbreak, before germ theory was established and before his waterborne theory had been formally accepted. He had been right. He had done everything a careful empirical reasoner should do but he spent the last years of his life largely ignored. The process by which expertise generates and revises knowledge is messier, slower, and more socially conditioned than the clean image of scientific method suggests. Understanding that process gives you better grounds for evaluating scientific claims.

A Spotter's Guide to Informal Fallacies

Paradigm Shift
Social Science Explainer

The Big Ideas in this lesson describe how good epistemological reasoning works - and where it breaks down. Bad arguments fail for multiple reasons: evidence may be incomplete, paradigms may be entrenched, and the reasoning may contain specific structural errors - fallacies - that make arguments look more convincing than they are. This is essential preparation for the Social Sciences, where contested claims are everywhere and the quality of the reasoning behind them is always the first question to ask.

One observation worth making: several of these fallacies work together in practice. A single paragraph of motivated reasoning often contains a post hoc claim, a hasty generalisation, a circular structure, and an ad hominem attack simultaneously. The ability to identify them separately - to name precisely which step is fallacious and why - is what transforms vague unease about an argument into a specific, defensible critique. That is the difference between knowing something is wrong and being able to say why.

Next: Lesson 6 - Methods and Tools 2 - Can we trust our own thinking?

Questions, assessments, films and other stuff.

Questions to think about

The early mask guidance was wrong. Was it also unjustified - or was it a justified claim that turned out to be false? Is there a difference, and does it matter?
Snow used a map; the miasmatists used mortality tables. Both were working with the same deaths. Did the tool change what the evidence could show - and does the choice of tool ever determine what a method can find?
Snow's investigation succeeded because he looked for cases that should have appeared on the map but did not. Is looking for disconfirming evidence a general obligation for any knower, or is it specific to scientific reasoning?
Hume argued that induction cannot be logically justified, yet we cannot do without it. Does this mean that all empirical knowledge is, at bottom, a matter of habit rather than reason?
Popper argues that the more a theory forbids, the better it is. Does this criterion apply outside the natural sciences - in history, in ethics, in economics? If not, what replaces it?
Kuhn and Popper disagreed about how science actually works. Popper believed scientists should try to falsify their theories; Kuhn observed that scientists rarely do this in practice. Which account is more epistemologically significant - the normative account of how science should work, or the descriptive account of how it does?
The early Covid guidance on masks was partly strategic as well as scientific. When an institution gives guidance that is both a knowledge claim and a policy decision, how should we evaluate whether it was justified?
Peer review, replication, and open publication are described as social tools of knowledge production. What happens to scientific knowledge when those tools malfunction - when peer review is captured by vested interests, replication is too expensive, or publication is paywalled?

Exhibition connections
See more exhibition ideas and previous student work here

It is never too early to start to think about your TOK Exhibition, the ideas in this lesson connect strongly to three of the 35 prompts. Start noticing objects in the world around you that speak to these questions.

Prompt #5: What counts as good evidence for a claim?
The lesson's central question. Snow's pump map was striking but insufficient on its own - the workhouse and brewery anomalies were what made the evidence compelling. The early Covid mask guidance was also based on available evidence; when that evidence was inadequate, the guidance failed. An exhibition object could be any piece of evidence whose meaning depends on the questions asked of it: a dataset, a map, a photograph, a scientific study. The connection lies in examining what makes it count as evidence for a particular claim - and what would count as evidence against it.

Prompt #19: What counts as a good justification for a claim?
Justification goes beyond evidence: it is the process of showing that the evidence supports the conclusion, by appropriate reasoning, to a sufficient degree. Popper's falsificationism gives one answer: a claim is well-justified if it has survived serious attempts to refute it. The mask guidance had not been tested against the superspreader evidence when it was issued. An exhibition object connected to a claim that has been tested and survived - a vaccine, a public health intervention, a historical account - allows exploration of what justification actually consists of.

Prompt #31: How can we judge when evidence is adequate?
Hume's problem means there is no formal answer to this question. In practice, adequacy is judged by communities of knowers applying shared standards - standards that are paradigm-dependent. The miasmatists and Snow were applying different standards for adequacy; both were doing so sincerely. The same was true of aerosol and droplet researchers in 2020. An exhibition object connected to a disputed question where evidence exists but is contested - climate science, a historical controversy, a public health debate - connects directly to the question of who decides when evidence is adequate, and on what grounds.

Feature films
For more see my 10 films for the TOK journey page.

🎬 WATCH — Contagion (2011)
Directed by Steven Soderbergh

An unusually accurate dramatisation of epidemiological investigation. The scientists tracing the MEV-1 transmission chain are doing exactly what Snow did - mapping cases, looking for anomalies, testing hypotheses against disconfirming evidence. Made before Covid-19 and eerily prescient about it: the film's depictions of institutional resistance, public misinformation, and the gap between what scientists know and what governments communicate became a reference point during the pandemic. The methodology scenes - particularly those involving Laurence Fishburne and Kate Winslet - directly illustrate the lesson's four big ideas. My students can watch the film here.

🎬 WATCH — Totally Under Control (2020)

Directed by Alex Gibney

A documentary examining the US government's response to Covid-19 in the first year of the pandemic, with particular attention to the gap between available evidence and official guidance. Gibney's method - interviewing scientists, public health officials, and policymakers - makes the epistemological questions visible: when did the evidence change? When did the guidance change? Why was there a lag? These are Kuhn's questions about paradigm resistance made concrete and recent. (Available on Amazon and Apple)

The Core Theme - Knowledge and the Knower

Lesson 5 - What counts as evidence? (Methods and Tools 1)

THE PROVOCATION

The Mask Reversal

Dr. Anthony Fauci, March 2020 60 Minutes

Before we begin: The language of arguments

Four big ideas about how we claim to know what we know.

​Big idea 1 - Evidence does not speak for itself

"He reviewed Farr's list again, looking this time for telltale absences." Steven Johnson, The Ghost Map (2006)

A guided tour of the Ghost Map Steven Johnson - TED

Big idea 2 - Induction: the engine of knowledge, and its problem

Big idea 3 - Popper's response: the logic of falsification.

Karl Popper's Falsification BBC Radio 4

Big idea 4 - The social life of knowledge

Paradigm Shift Social Science Explainer

Bringing it together

A Spotter's Guide to Informal Fallacies

Paradigm Shift Social Science Explainer

Questions, assessments, films and other stuff.

Feature films For more see my 10 films for the TOK journey page.

Dr. Anthony Fauci, March 2020
60 Minutes

Big idea 1 - Evidence does not speak for itself

"He reviewed Farr's list again, looking this time for telltale absences."
Steven Johnson, The Ghost Map (2006)

A guided tour of the Ghost Map
Steven Johnson - TED

Karl Popper's Falsification
BBC Radio 4

Paradigm Shift
Social Science Explainer

Paradigm Shift
Social Science Explainer

Feature films
For more see my 10 films for the TOK journey page.