Bayesian statistics treats probability as a degree of belief, updated as evidence arrives.
The Core Idea
Start with a prior belief, observe data, and combine them to get a posterior belief. Bayes' theorem does the combining.
An Example
A test for a rare condition is 95% accurate. If only 1 in 1,000 people have the condition, a positive result still means the person probably doesn't have it — because false positives from the many healthy people outnumber true positives. Ignoring base rates is a common reasoning error.
Why Analysts Use It
- Natural statements: "there's an 85% probability variant B is better".
- Incorporates prior knowledge.
- Works well with small samples.
- Produces full distributions of uncertainty.
Bayesian A/B Testing
Rather than p-values, estimate the probability each variant is best and the expected loss of choosing it.
Cautions
- Priors should be justified and checked for sensitivity.
- Computation can be heavier, though modern tools make it accessible.
A Habit of Mind
Even without formal models, thinking in base rates and updating on evidence improves judgement.