6. Can we trust our own thinking? | Quomodo Scimus : IBTOK

The Core Theme - Knowledge and the Knower

Lesson 6 - Can we trust our own thinking? (Methods and Tools 2)

Knowing about a bias is not the same as being free of it. This lesson asks why, and what, if anything, can be done

THE PROVOCATION

You Already Know This. So Why Does It Keep Happening?

If you already knew about the gorilla, you see it. That is exactly the point. The gorilla test is about attention, not bias - what you look for determines what you find. The bat and ball is the harder problem. You know the answer is not 10 cents. You still feel that it is.

Critical Thinking 11e

This is the film we began the Critical Thinking course with in 11e. Watch it again.

It is a test made up of 5 questions: a simple visual illusion, the world's shortest IQ test (Cognitive Reflection Test - CRT) and the famous Daniel Simons and Christopher Chabris 'Invisible Gorilla' selective attention test.

The gorilla walked through the middle of the screen. About half of all first-time viewers miss it entirely, not because they are unobservant, but because System 2 has taken over. Counting passes is a deliberate, effortful task that demands focused attention. While System 2 is fully engaged on that task, System 1's broader awareness is suppressed. If you already knew about the gorilla, you spotted it immediately - because you were not running a System 2 counting task.

The bat and ball question is a different and more uncomfortable problem. The answer that comes to mind immediately is 10 cents. It feels right. It arrives with confidence. Research consistently shows that only a minority of people answer correctly without being prompted to slow down and check. Here System 1 is the problem: it generates a fast, confident answer - 10 cents - before System 2 has been asked to check it. The correct answer is 5 cents, but even students who know about System 1 and System 2, who have studied Kahneman, who have thought carefully about cognitive bias, still often feel the pull of the wrong answer. Knowing about it does not fix it. System 2 has to be deliberately engaged, and it resists. This is the harder problem - and it is what this lesson is really about.

You encountered these ideas in 11e and should know the vocabulary. The harder question is why knowing about System 1 does not protect you from it. The answer turns out to require not just a better account of how individual minds work, but a rethinking of what 'reasoning' is for in the first place.

Four big ideas about how the mind actually works.

Big idea 1 - The mind you cannot see running

You have two cognitive systems. The fast one is not just quick - it is invisible to itself. And that invisibility is the source of most of the problems.

"System 1 does not keep track of alternatives that it rejects, or even of the fact that there were alternatives. Conscious doubt is not in the repertoire of System 1; it requires maintaining incompatible interpretations in mind at the same time, which demands mental effort. Uncertainty and doubt are the domain of System 2."
Daniel Kahneman, Thinking, Fast and Slow (2011)

System 1 operates automatically, continuously, and outside conscious awareness. It is responsible for recognising faces, reading emotions, understanding sentences, driving on a familiar road, and producing the answer 10 cents when you read the bat and ball question. It does all of this faster than consciousness can follow, and it does not report what it is doing or why. By the time you are aware of having reached a conclusion, System 1 has already reached it.

System 2 is the deliberate, effortful, sequential reasoning you use when you check your working in a maths problem, or when you navigate an unfamiliar city. It is slower, more reliable, and dramatically more expensive in terms of mental energy. It is also, critically, lazy. System 2 does not engage unless it has to. Most of the time, it endorses whatever System 1 has already produced.

In the interview below, Kahneman explains the distinction in his own words - and makes the point that is hardest to grasp in writing: knowing you are in a cognitive illusion does not make it feel less real.

Daniel Kahneman: Thinking Fast and Slow Inc.

Daniel Kahneman (1934-2024) was Professor Emeritus of Psychology at Princeton, a psychologist awarded the Nobel Prize in Economics in 2002 for his work with Amos Tversky on judgement and decision-making. In this six-minute interview he explains System 1 and System 2 and draws a crucial distinction between visual illusions and cognitive illusions. Watch particularly from 3:30, where he describes why knowing about a cognitive error does not stop it feeling correct - and the final exchange on whether gut instinct is reliable.

The System 1 / System 2 architecture creates a specific and serious problem. System 1 does not monitor or evaluate its own outputs. It does not flag its conclusions as provisional or as generated by fast pattern-matching rather than careful analysis. When you read the bat and ball question, System 1 does not say "this is the kind of question where I tend to get the wrong answer, you should check this one carefully". It just says: 10 cents. This is why knowing about System 1 does not reliably fix it.

The bias operates in System 1, before System 2 has been consulted. To override it, System 2 would have to engage on every question, treating every automatic response as provisional and requiring verification. That is cognitively exhausting and, in practice, impossible. We cannot reason about everything from scratch. The architecture that creates our vulnerabilities is also the architecture that makes everyday cognition possible at all. Kahneman and his collaborator Amos Tversky spent decades mapping the specific patterns in which System 1 misleads. Three of these - anchoring, availability, and representativeness - are worth knowing by name, because once you have the name you start seeing the pattern everywhere.

Big idea 2 - Fast thinking is not always wrong

Kahneman's framework is essential. But it has a bias of its own: it makes System 1 look like a defect. Gigerenzer shows that heuristics evolved because they work.

"Gut feelings are based on surprisingly little information. That makes them look untrustworthy in the eyes of our superego, which has internalized the credo that more is always better. Yet experiments demonstrate the amazing fact that less time and information can improve decisions." Gerd Gigerenzer, Gut Feelings (2007)

Gerd Gigerenzer spent much of his career in productive disagreement with Kahneman. His argument is not that cognitive biases do not exist, it is that the standard account systematically undervalues what fast, intuitive thinking actually achieves, and misidentifies what counts as a good decision. A heuristic - a fast, simple rule for making decisions - is not good or bad in the abstract. It is good or bad relative to the environment in which it operates. As we saw back in 11e, a master chess player who sees a board and immediately senses a threat is not bypassing cognition - they are deploying a vast library of pattern-matching built through thousands of hours of deliberate practice. A firefighter who senses that a burning room is about to collapse, without being able to articulate why, is reading real signals from a trained perceptual system. In both cases, the fast response is more reliable than slow analysis would be, because the expert has developed System 1 responses that are calibrated to the domain. Research on stock traders, by contrast, finds that experienced traders do not consistently outperform novices or statistical models, because financial markets are irregular enough that feedback is too noisy to calibrate intuition reliably.

Heuristics and biases
Learn Liberty

Sean Rife explains that heuristics are mental shortcuts developed from experience that help us make quick decisions without constant deliberation. They are efficient but not designed for accuracy, which can lead to systematic errors. Examples like the availability heuristic (overestimating violence) and representativeness heuristic (misjudging medical probabilities) show how intuition can mislead, even experts. Heuristics also shape political beliefs. While necessary, they require humility, recognising their limits and open to correction in an uncertain world.

Two bestselling books extend Gigerenzer's argument in different directions. As we have seen in Big Idea 2 in Lesson 3, Malcolm Gladwell's Blink (2005) tells the same story through cases: the art experts who sensed a forgery before they could say why, the psychologist who could predict divorce from a fifteen-minute conversation, the military commander who won a war-game simulation by trusting instinct over data models etc. (See further reading at the end of this page)

Richard Thaler and Cass Sunstein's Nudge (2008) takes the argument in a different direction entirely. If System 1 is shaped by the environment it operates in, then redesigning that environment - the choice architecture - can steer heuristics toward better outcomes without restricting freedom. The most studied example: changing organ donation registration from opt-in to opt-out dramatically increases donor rates, not because anyone's values change but because the default activates a different System 1 response. The same tool is used in pension auto-enrollment, food placement in canteen queues, and other choice environments. Choice architecture is an intervention tool - one that compensates for System 1's susceptibility to framing by deliberately controlling the frame. Governments quickly adopted the approach: the UK established the Behavioural Insights Team in 2010 (often called the "Nudge Unit"), applying behavioural tweaks to increase tax compliance, organ donation, and savings rates. Similar units spread to the US, EU, and beyond during the 2010s. This is a constructive answer to the problem Kahneman diagnosed.

iNudgeyou - inspired by Richard Thaler and Cass Sunstein
Inc.

Nudge presents itself as libertarian paternalism: it preserves freedom of choice while designing environments that make better choices easier. Critics argue this is still manipulation - that controlling the frame without disclosing you are doing so exploits exactly the System 1 mechanisms that make people vulnerable. Thaler and Sunstein's response is that all environments have a choice architecture whether you design it or not: putting the fruit at eye level in a cafeteria is a choice, and so is not doing so. The question is who controls it and toward what ends.

iNudgeyou is a Danish behavioural insights organisation based in Copenhagen that applies "nudge" theory to public policy and organisational change. Founded by Pelle Guldborg Hansen, it works with governments and institutions to design interventions that subtly shape behaviour without restricting choice. Projects include improving health decisions, increasing compliance, and promoting sustainable habits through changes in messaging, defaults, and environments. It represents the global spread of behavioural policy approaches inspired by Richard Thaler and Cass Sunstein. Watch these two short, brilliant examples below.

Big idea 3 - Ten thousand hours - and what the science actually says

Almost everyone has heard the rule. Almost nobody knows what it actually says. The gap between the popular version and the research is itself a lesson in how knowledge travels - and what gets lost on the way.

"Robert Bjork's idea of the sweet spot: that productive, uncomfortable terrain located just beyond our current abilities, where our reach exceeds our grasp."
Daniel Coyle, The Talent Code (2009), describing Robert Bjork's concept

The ten thousand hours rule is one of the most widely cited findings in popular psychology. Malcolm Gladwell introduced it in Outliers (2008): to achieve world-class expertise in any domain, you need approximately ten thousand hours of practice. The rule migrated quickly into business advice, sports coaching, education policy, and everyday conversation. The formula as it entered popular culture is a significant simplification of the research - and of what Gladwell himself had meant to say.

Malcolm Gladwell on the 10,000 hours rule
MasterClass

Malcolm Gladwell is a bestselling British-Canadian journalist known for turning social science research into accessible narratives. He popularised the idea of the "10,000 hours" rule in Outliers, suggesting expertise comes from long-term deliberate practice rather than innate talent. His major books include Outliers, The Tipping Point, Blink, and Talking to Strangers. Across his work he explores how psychology, sociology and chance shape success, decision-making and everyday behaviour, often challenging conventional assumptions.

The basis of the 10,000 hours research comes from K. Anders Ericsson, a Swedish psychologist who spent four decades studying expertise across chess, music, medicine, and sport. Ericsson's central finding was that exceptional performers had accumulated large amounts of practice - but the critical variable was quality, not merely quantity. What distinguished the best from the very good was the character of their practice: structured, effortful sessions at the boundary of current ability, with immediate corrective feedback. Ericsson called this deliberate practice - and he was clear that most practice is not deliberate. Deliberate practice means drilling on precisely the passages where you make errors. Playing through a piece you already know fluently reinforces existing patterns; it builds habit without expanding capability. Ten thousand hours of practice that stays within the comfort zone accumulates time without producing expertise.

Daniel Coyle's The Talent Code (2009) adds the neuroscience to the behavioural psychology. The mechanism behind deliberate practice is myelination: deep, effortful practice triggers the growth of myelin, the insulating sheath around neural circuits that increases the speed and accuracy of signals. The struggle at the boundary of your ability - making errors, correcting them, pushing into unfamiliar territory - is the biological stimulus for myelin growth, where smooth, comfortable practice within existing ability produces none. This is why Coyle emphasises difficulty: the discomfort at the edge of ability is the mechanism through which skill develops.

There is a version of this that applies to you, the student, reading this lesson right now. The standard approach to revision - re-reading notes, highlighting, reviewing familiar material - is comfortable and feels productive. Research consistently shows it is among the least effective study strategies. Retrieval practice - testing yourself, working problems from scratch, attempting material before you feel ready - is effortful, uncomfortable, and significantly more effective. In cognitive terms: building reliable System 1 responses in any domain requires first working through that domain explicitly with System 2. The expert's intuition is the outcome of that deliberate process.

The tools of deliberate practice
TED-Ed

Ericsson's research identified a set of specific practice tools that produce expertise: working at the outer edge of current ability (not within the comfort zone); immediate corrective feedback (knowing within seconds whether a move or a diagnosis was right); structured repetition of specific weaknesses rather than general rehearsal; and a teacher or coach who can identify the gap between current and target performance. Each is a procedural requirement. Without all four, hours of practice accumulate without producing expertise.

The implication is uncomfortable: what feels like natural, comfortable, engaging learning may not be the most effective route to genuine competence. This connects the expertise research back to Gigerenzer's point about regular environments. Expertise is exactly what makes an environment regular for a knower. The chess grandmaster's System 1 is reliable because deliberate practice has built the pattern library it draws on.

Big idea 4 - Reason evolved to win arguments, not find truth

Now the deepest explanation. Reason is working exactly as it evolved to work - and what it evolved for turns out to be something other than truth.

"Just as echolocation evolved as an adaptation to the ecological niche inhabited by bats, reason evolved as an adaptation to a very special ecological niche, a niche that humans built and maintain for themselves with their intense social relationships, powerful languages, and rich culture." Hugo Mercier and Dan Sperber, The Enigma of Reason (2017)

Hugo Mercier and Dan Sperber's argumentative theory of reason is the most radical rethinking of reasoning in recent cognitive science. Their central claim is that confirmation bias is what reason was designed to do. Reason evolved as a tool for social argumentation: producing justifications, challenging other people's claims, and defending your own position under pressure. The evidence for this is striking. People are dramatically better at finding flaws in other people's reasoning than in their own. Give someone a weak argument in support of a position they hold, and they will defend it. Give them the same argument in support of a position they oppose, and they will immediately identify its weaknesses. What drives the evaluation is whose side the argument is on; logical quality is a distant second. This is myside bias, and it is one of the most revolutionary findings in the psychology of reasoning.

The argumentative theory accounts for this pattern: a system built for social persuasion would produce exactly this response profile. In a group where disagreement and criticism are genuinely present, the overall system can work quite well. Bad arguments get challenged, weak positions get exposed, and the group converges on better answers than any individual would reach alone. The reliability of reason is a collective achievement.

This explains why knowing about your own biases does not fix them. Confirmation bias is the argumentative function of reason doing what it was built to do. Self-monitoring cannot change that. What works is exposure to genuine challenge from people who do not share your starting position. This is one reason why good epistemology is social. Science works because the institution creates systematic pressure to challenge and refute. Peer review, replication, and open publication are mechanisms that compensate for individual argumentative bias, precisely because they force claims in front of people with no stake in confirming them. This is what IB means by the community of knowers: reliable knowledge is built through social structures - institutions designed to expose claims to challenge.

It also explains something from [Lesson 4]. Manufactured doubt works so reliably because it exploits the architecture of argumentative reason. Introduce enough counter-arguments, even weak ones, and the myside bias of people who already hold the challenged view kicks in. They defend and dig in. The goal of manufactured doubt is to trigger the defensive function of reason in people who need to be kept uncertain. Persuasion is beside the point.

Big idea 5 - The elephant decides. The rider explains.

Jonathan Haidt applies everything in this lesson to the domain where people most believe they are reasoning carefully: moral judgment.

"The rider is our conscious reasoning - the stream of words and images of which we are fully aware. The elephant is the other 99 percent of mental processes - the ones that occur outside of awareness but that actually govern most of our behavior."
Jonathan Haidt, The Righteous Mind (2012)

Jonathan Haidt's central claim in The Righteous Mind challenges the standard picture of moral judgment. Moral conclusions arrive through fast, automatic intuition - and then we produce reasons to justify them after the fact. The reasons feel like causes. They are, mostly, post-hoc rationalisation.

Haidt's evidence includes a series of moral dumbfounding experiments - a specific research tool designed to expose the gap between moral intuitions and moral reasoning. Participants are presented with scenarios that trigger strong moral intuitions but for which the standard moral arguments - harm, unfairness - do not apply. The classic case: a brother and sister who, on one occasion during a holiday, decide to have sex. They use contraception, no one is harmed, no one finds out, and they both agree it brought them closer. Is this wrong? Almost everyone says yes - immediately and with certainty. When pressed for reasons, they cite harm or the risk of genetic problems. When these are ruled out by the scenario, they continue to insist it is wrong but cannot say why. They have reached a conclusion through intuition and cannot find a logical justification for it.

Jonathan Haidt - The Righteous Mind
Rare

Jonathan Haidt is an American social psychologist known for his research on moral psychology, political division, and the foundations of moral reasoning. He is a professor at New York University Stern School of Business and has influenced debates on ideology and culture. His book The Righteous Mind argues that morality is intuitive, shaped by evolution, and varies across cultures, explaining political disagreement. It explores why people prioritize loyalty, authority, and fairness differently in moral judgement.

The rider-elephant metaphor captures the relationship precisely. The elephant (System 1, the emotional and intuitive system) moves where it wants to go. The rider (conscious reasoning) tells a story about why the elephant is going in this direction. The story is coherent. It may even be persuasive. But it did not cause the movement. This is why moral arguments between people who start from different intuitions so rarely change minds: the rider on each side is defending a position the elephant has already taken - the question is settled before the argument begins. Reasoning can still influence others, and over time reshape our own intuitions. The mechanism is slow and social: accumulated exposure and experience shift intuitions gradually, ahead of any argument that articulates the change.

Haidt's position is that reasoning rarely changes your own mind. It can change other people's intuitions, but only when delivered in the right relational context. The social intuitionist model proposes that moral progress happens through the kinds of contact and experience that shift intuitions before reasons are constructed. Lessons 7 and 8 take that claim seriously.

Bringing it together

Understanding the architecture is where improvement begins

This has been a very long lesson, with a lot going on. But remember this is my TOK website. About 50% of all the non-fiction I have read in the last 10 years has been a follow up to my first reading of Daniel Kahneman's Thinking Fast and Slow. I used to teach TOK like the former philosophy graduate I was, now I routinely refer to the physical parts of the brain. For a history teacher this is somewhat disconcerting. In brief, trust me, this page could have been a lot longer.

Lesson 5 showed how knowledge is supposed to be justified - through evidence, induction, falsification, and scientific reasoning. This lesson has shown why meeting those standards is genuinely difficult. System 1 operates before System 2 is consulted. Heuristics are often reliable, but not in the ways we think. Expertise requires deliberate practice - and most of us do not practise deliberately. Reason evolved as a tool for argumentation rather than truth-finding. And even our moral reasoning, which feels most deliberate and principled, is mostly the rider explaining where the elephant has already gone.

Stanislas Dehaene, the French cognitive neuroscientist, offers a constructive answer in How We Learn. His four pillars of effective learning are a toolkit for building better cognitive habits: attention (focused engagement without distraction), active engagement (generating material rather than passively receiving it), error feedback (knowing quickly when a response was wrong, so System 2 can correct before System 1 embeds the error), and sleep-based consolidation (the process by which the day's learning is transferred from working memory to long-term storage overnight). Ericsson's deliberate practice and Coyle's myelin story point at the same underlying process that Dehaene's four pillars map from neuroscience. Skill is System 2 discipline, accumulated and internalised until it becomes System 1 fluency. The chess grandmaster's fast, reliable intuition was once slow, effortful, and frequently wrong. The expert's tools began as slow thinking, repeated until the underlying circuits were built and speed followed.

stanislas-dehaene-four-pillars-of-effective-learning.webp

The answer to cognitive bias is building better habits of thought through practice and feedback, in social environments that provide genuine challenge. Permanent vigilance is exhausting and misses the structural point: bias operates before System 2 is consulted. Science, peer review, structured argument, and education are social technologies for compensating for the limits of individual cognition. Their function is to create conditions under which bias can be checked, corrected, and gradually reduced over time. It is an imperfect answer - but, Dehaene suggests, the most we can honestly claim.

Next: Lesson 6 - Methods and Tools 2 - Can we trust our own thinking?

Questions, assessments, films and other stuff.

Questions to think about

If knowing about a bias does not reliably protect you from it, what exactly is the value of studying cognitive science for a TOK student?
Gigerenzer argues that heuristics are ecologically rational - good relative to their environment. Does this mean that "irrational" thinking is sometimes the most rational response to a situation?
The cognitive scientists' main tools - controlled experiments, reaction time measurements, fMRI scanning - study the mind under artificial conditions. Do they tell us how biased ordinary reasoning is, or only how biased it is when we are deliberately trying to trigger errors?
Ericsson argues that expertise is built through deliberate practice, not innate talent. If this is correct, what are the implications for how schools assess and reward ability - and for what counts as a fair educational system?
The ten thousand hours rule lost its most important word - "deliberate" - when it moved from research into popular culture. What does this suggest about the relationship between expert knowledge and publicly available knowledge?
Choice architecture (nudge) steers decisions by exploiting known System 1 responses. Is this ethically equivalent to providing better information - or is it a different kind of epistemic intervention?
Mercier and Sperber say reason evolved to win arguments, not find truth. If this is correct, what does it imply about the reliability of individual reasoning versus collective reasoning under conditions of genuine disagreement?
Haidt shows that people reach moral conclusions before they can articulate reasons for them. Does this mean that moral reasoning is an illusion - or does it mean something more specific about the relationship between intuition and justification?
If the rider mostly explains what the elephant has already decided, can moral argument ever genuinely change minds? Under what conditions might it do so?
Dehaene argues that expertise is trained intuition - System 2 becoming System 1 through practice. Does this mean that the distinction between fast and slow thinking is not fixed but developmental?

Exhibition connections
See more exhibition ideas and previous student work here

It is never too early to start to think about your TOK Exhibition, the ideas in this lesson connect strongly to three of the 35 prompts. Start noticing objects in the world around you that speak to these questions.

Prompt 12: Is bias inevitable in the production of knowledge?

Mercier and Sperber's argumentative theory gives this question a precise answer: myside bias is a structural feature of how reason works. An exhibition object connected to a knowledge claim that reflects the interests or prior commitments of its producer - a corporate-funded study or a politically motivated report - allows exploration of whether the bias is accidental or systematic, and whether systematic bias can ever be fully corrected or only acknowledged and managed.

Prompt 26: Does our knowledge depend on our interactions with other knowers?

This is Mercier and Sperber's central claim in institutional form. Reason achieves reliability through dialogue: the myside bias of each individual is checked by the challenges of others. An exhibition object connected to a collective knowledge-producing institution - a scientific journal, a court of law, a fact-checking organisation, a Wikipedia article - allows exploration of how the social structure of the institution compensates for individual cognitive limits, and what happens when those social structures break down.

Prompt 28: To what extent is objectivity possible in the production or acquisition of knowledge?

Kahneman's WYSIATI principle - what you see is all there is - means that individual knowers cannot even perceive the alternatives they are not considering. Haidt's rider-elephant model means that even careful moral reasoners are mostly post-hoc rationalising. Objectivity for an individual is severely constrained. An exhibition object that represents a contested knowledge claim - in science, history, or ethics - opens the question of whether objectivity is a property of individual knowers, of methods, or of institutions, and whether any of these can fully achieve it.

Feature films
For more see my 10 films for the TOK journey page.

🎬 WATCH — Inside Out (2015)
Directed by Pete Docter

Not a science documentary, but one of the most accurate visualisations of what Kahneman, Haidt, and the cognitive scientists are describing. The film dramatises the relationship between emotion (the elephant) and reasoning (the rider), showing that the emotional system is not an obstacle to good thinking but its foundation. The key scene is when Joy learns that Sadness has a necessary function - that some knowledge requires negative emotion to be processed at all. My students can watch the film here.

🎬 WATCH — Behind the Curve (2018)

Directed by Daniel J. Clark

A documentary about the flat earth movement that becomes, accidentally, one of the best illustrations of Mercier and Sperber's argumentative theory of reason ever committed to film. The flat earthers are not stupid. They reason carefully, conduct experiments, cite evidence, and build community. What the film shows is exactly what the theory predicts: reason deployed in service of a prior commitment, with every anomalous result reinterpreted rather than treated as refutation. The film's most famous scene - in which flat earthers conduct a laser experiment that produces results inconsistent with flat earth theory, and immediately begin explaining why the result must be wrong - is Kuhn and Mercier and Sperber in the same moment. Connects directly to Lessons 5 and 6. My students can watch the film here.

🎬 WATCH — Don't Look Up (2021)

Directed by Adam McKay

A satirical film about two astronomers who discover a comet on a collision course with Earth and find that nobody wants to act on the evidence. Heavy-handed as satire, but useful as an illustration of motivated reasoning and the way that political identity, economic interest, and media dynamics can shape what people allow themselves to believe. The film is most useful for the Knowledge Questions on collective reasoning and institutional design: the scientists are right, the evidence is clear, and it changes nothing. Why not, and what would have needed to be different? My students can watch the film here.