A hypothesis test asks whether an observed pattern is likely to be real or could easily have happened by chance.
The Setup
- Null hypothesis (H0): no effect or no difference — the new page converts the same as the old.
- Alternative hypothesis (H1): there is an effect.
- Collect data and compute a test statistic.
- The p-value is the probability of seeing a result at least this extreme if the null hypothesis were true.
A small p-value (commonly below 0.05) suggests the data is unlikely under the null, so you reject it.
What a p-Value Is Not
- Not the probability that the null hypothesis is true.
- Not a measure of how large or important an effect is.
- Not proof — just evidence under assumptions.
Errors
- Type I error (false positive): rejecting a true null. Controlled by the significance level.
- Type II error (false negative): missing a real effect. Reduced by larger samples.
Common Tests
- t-test: compare means of two groups.
- Chi-squared test: compare proportions or check association between categories.
- Mann–Whitney U: compare two groups without assuming normality.
- ANOVA: compare means across several groups.
Good Practice
- Decide the test and sample size before collecting data.
- Report effect sizes and confidence intervals, not just p-values.
- Beware multiple comparisons: test enough things and some will look significant by chance.
- Practical significance matters as much as statistical significance.