Multiple Testing and False Discoveries

Testing many hypotheses at once inflates the chance of a false positive; Bonferroni correction controls it at the cost of power.

1. The idea

If you run $m$ independent hypothesis tests each at significance level $α$ , the probability of at least one false positive is $1 - (1 - α)^{m}$ , which grows quickly with $m$ even when $α$ is small.

The Bonferroni correction controls this by testing each hypothesis at level $α / m$ instead of $α$ , guaranteeing the overall false-positive probability stays at or below $α$ (via a union bound), at the cost of reduced power to detect true effects.

In quant research this shows up as backtest overfitting: testing thousands of trading signals and reporting only the significant ones systematically overstates how many are real, since some will appear significant purely by chance.

2. Why it matters

Understanding why backtesting many strategies inflates false positives — and how to correct for it — is one of the most heavily tested statistical-reasoning topics in quant research interviews.

3. Core mathematics

Chance of ≥1 false positive

P (at least one false positive) = 1 - (1 - α)^{m}

Assuming m independent true-null tests.

Bonferroni-corrected threshold

α_{test} = \frac{α}{m}

4. How to recognize it

Think of this when…

A strategy or signal was chosen as 'the best' out of many candidates tested
The question mentions backtesting many parameter combinations or trading rules
You need to adjust a p-value threshold for the number of comparisons made

5. Worked example

Worked example

A researcher tests 100 independent trading signals, each with a true null hypothesis, at significance level $α = 0.05$ . What is the approximate probability at least one appears significant by chance? What threshold should be used per test to keep the overall false-positive rate at 0.05?

1
Compute the chance of at least one false positive out of 100 independent tests.
$P (at least one) = 1 - (0.95)^{100}$
2
Evaluate numerically: $(0.95)^{100} \approx 0.0059$ .
$P (at least one) \approx 1 - 0.0059 = 0.994$
3
Apply Bonferroni correction to control the overall rate at 0.05.
$α_{test} = \frac{0.05}{100} = 0.0005$
Answer: ≈99.4% chance of at least one false 'discovery' uncorrected; use α = 0.0005 per test with Bonferroni

6. Common mistakes

Reporting a single strategy's p-value from a large search without correcting for the number of strategies tried
Applying Bonferroni when tests are highly correlated, making it overly conservative
Confusing the corrected per-test threshold with the overall family-wise error rate
Ignoring multiple testing entirely when a backtest optimizes over many parameters

7. Quick check

Why does testing 500 trading strategies and reporting only the significant ones overstate real effects?

8. Practice

2 QuantSolve questions use this concept.

Practice this concept

9. Related methods

This lesson is conceptual; see the topic hub for methods.

Create a free account to save lesson progress.