Skip to content

Multiple Comparisons and False Discoveries

Why testing many hypotheses produces false positives, and corrections that keep conclusions trustworthy.

Editorial team 1 min read

Test enough things and some will appear significant by chance. This is the multiple comparisons problem.

The Problem

At a 5% significance level, testing 20 unrelated metrics with no real effects will, on average, produce one "significant" result.

Where It Happens

  • A/B tests with many metrics or segments.
  • Exploring many features for correlations.
  • Checking results repeatedly during a test.
  • Trying many analyses until one works — sometimes called p-hacking.

Corrections

  • Bonferroni: divide the significance level by the number of tests. Simple and conservative.
  • Holm: a less conservative step-down version.
  • False discovery rate (Benjamini-Hochberg): controls the proportion of false positives among discoveries; good for exploratory work.

Better Practice

  • Declare primary metrics and hypotheses in advance.
  • Treat unplanned findings as hypotheses to test again.
  • Report how many comparisons were made.
  • Replicate important results.

More in Data science & analytics

All Data science & analytics guides →
Data science & analytics Guide · 2 min

Descriptive Statistics Essentials

Mean, median, mode, spread and shape: the summary numbers every analysis starts with, and when each one misleads.

Data science & analytics 2 min read 6 Mar 2026

Data science & analytics Guide · 2 min

Probability Basics for Data Work

The probability ideas analysts use every day: events, conditional probability, independence and Bayes' theorem.

Data science & analytics 2 min read 5 Mar 2026

Data science & analytics Guide · 2 min

Common Probability Distributions

Normal, binomial, Poisson, exponential and more: recognising the shapes data takes and what they imply.

Data science & analytics 2 min read 4 Mar 2026

Data science & analytics Guide · 2 min

Hypothesis Testing Explained

Null hypotheses, p-values and significance: what a hypothesis test tells you, and the misunderstandings to avoid.

Data science & analytics 2 min read 3 Mar 2026