Treating small samples as equally reliable as large ones when judging probabilities, ignoring how much randomness small samples contain.
Imagine you flip a coin 4 times and get 3 heads. You might think 'wow, this coin lands on heads 75% of the time!' But if you flipped it 1,000 times, you'd get much closer to 50/50. We forget that tiny amounts of information can look very different from the big picture, like tasting one grape and deciding the whole vineyard is sour.
Insensitivity to sample size occurs when people fail to appreciate that smaller samples are inherently more variable and less reliable than larger ones, leading them to draw equally strong conclusions regardless of how many data points underlie a result. This bias manifests as treating a finding from 10 observations with the same confidence as one from 10,000 observations. People intuitively expect any random sample, no matter how small, to closely mirror the overall population — a misconception Tversky and Kahneman called 'belief in the law of small numbers.' The result is systematic overconfidence in patterns detected in limited data and a failure to demand more evidence before drawing conclusions.
The same glitch looks different depending on the terrain. Finance, medicine, a relationship, a team — same mechanism, different costume.
Investors frequently evaluate mutual funds or stock-picking strategies based on a few months or quarters of returns, treating short-term outperformance as evidence of genuine skill rather than recognizing that small time samples produce high variability and that streaks are expected by chance alone.
Clinicians may draw strong conclusions about a treatment's efficacy from a handful of patients they've personally treated, overriding large randomized controlled trials. Similarly, patients dismiss large-scale safety data in favor of a few anecdotal adverse reactions reported by people they know.
Teachers form confident judgments about a student's ability from one or two early assignments, and school administrators rank programs or curricula based on test scores from very small cohorts where random variation dominates actual quality differences.
People judge a new partner's character based on a few early interactions — a small sample of behavior — and form rigid expectations that resist updating even as more information becomes available over time.
Product teams run A/B tests with insufficient sample sizes and ship features based on early results that appear statistically significant, only to see the effect vanish when rolled out to the full user base. UX researchers draw design conclusions from usability tests with 3-5 participants without acknowledging the limits of such small samples.
Performance reviews are heavily influenced by a few recent memorable incidents rather than a full year of work. Hiring committees generalize about entire universities, bootcamps, or previous employers from encounters with just a few candidates.
Polls with very small sample sizes are reported with the same authority as large-scale surveys. Voters and commentators treat a handful of town hall reactions or viral social media posts as representative of public opinion at large.
Amos Tversky and Daniel Kahneman, 1971 ('Belief in the Law of Small Numbers,' Psychological Bulletin, 76(2), 105–110); further elaborated in their 1974 paper 'Judgment under Uncertainty: Heuristics and Biases' in Science.
In ancestral environments, humans rarely encountered formal statistical data. Decisions were made from small, personally observed samples — a few encounters with a predator, a handful of foraging trips to a location. Treating these small observations as representative was often adaptive because the cost of waiting for a larger sample (e.g., more predator encounters) could be fatal. Quick generalization from limited experience was a survival advantage when data collection itself was dangerous.
Machine learning models trained on small or unrepresentative datasets can produce overfit predictions that appear highly accurate during training but fail to generalize. Practitioners who evaluate model performance on small test sets may overestimate accuracy. In recommendation systems, items with few ratings can receive extreme average scores (very high or very low) that disproportionately influence recommendations, a problem known as the cold-start problem.
50 cards are free to preview. Buyers unlock the rest of the deck plus the interactive training — Spot-the-Bias Quiz unlimited, Swipe Deck with spaced repetition, My Blindspots, Decision Pre-Flight, the Printable Deck + Cheat Sheets, and the Field Guide e-book. $29.50$59.
