Multiple testing: when the best backtest is a lucky accident
Track the search process and separate a genuine hypothesis from the winner of a large historical contest.
One reported strategy can hide a thousand trials
A researcher may publish one attractive backtest after trying many signals, parameter values, assets and sample dates. The final chart does not show the full search. Even if every candidate has no real edge, the best-looking candidate can appear impressive because some random sequences are favorable.
The relevant unit of analysis is therefore not just the selected rule but the procedure that selected it. A threshold changed after inspecting results counts as part of the search even when no formal optimization program was run. Memory-based experimentation, discarded spreadsheets and informal visual inspection can all contribute to selection bias. An honest research record makes these choices visible.
Understand the simple probability illustration
If twenty independent tests each have a 5% false-positive probability, the chance of at least one false positive is 1 − 0.95²⁰, approximately 64.2%. The independence assumption is important: trading variants are often correlated, so this arithmetic is an illustration rather than a precise correction for an actual research program.
The lesson is not that every positive finding is false. It is that a single-test threshold cannot be interpreted the same way after a broad search. Related parameter choices do not create fully independent evidence, yet their correlation does not make selection bias disappear. Estimating the effective search breadth is itself uncertain and should not be disguised as an exact count.
A p-value is not the probability that a trading hypothesis is true, nor the probability that its next trade will win. It measures how incompatible a specified statistic is with a specified null model, under the test’s assumptions. A small number can reflect an invalid noise model, dependence that was ignored, or selective reporting as well as a genuine relationship. Statistical significance also says nothing by itself about whether the effect is large enough to cover spread, financing and implementation costs.
Protect the hypothesis from the scoring rule
Start with an economic rationale that specifies why a relationship might persist and why it might stop working. Define data cleaning, costs and evaluation criteria before looking at the result where feasible. This does not eliminate bias, but it narrows the freedom to retrofit a story around a fortunate pattern.
Keep a trial log containing rejected variants and the reason for each change. Distinguish debugging from hypothesis alteration. Correcting a timestamp error is different from moving a stop level because it improves the equity curve. Both changes should be recorded, since seeing results can influence later decisions even when the initial change was legitimate.
An untouched sample has a limited life
A holdout period is valuable only while it remains untouched by development. Once a researcher repeatedly consults it to choose among models, it becomes another training resource. Calling it out of sample does not reverse the information flow. A newer period or another genuinely independent evaluation may then be needed.
Walk-forward procedures, nested selection and research on backtest overfitting can help examine the selection process, but none converts weak data into certainty. Assess stability across sensible parameter neighborhoods and economically distinct periods. Stability is encouraging when it was evaluated honestly; searching for whichever stability statistic looks best introduces another layer of testing.
Reward evidence quality rather than the tallest chart
Compare results with a simple benchmark, realistic costs and uncertainty estimates. A complicated model that barely improves a noisy metric may not justify its additional degrees of freedom. Examine how much of the apparent advantage comes from one trade, one asset or one choice of starting date.
A useful report includes the search history, the final specification, failures and unresolved assumptions. Negative findings reduce the space of plausible ideas and can save future effort. The aim is not to preserve a favorite strategy but to discover whether the available evidence supports any claim strong enough to survive replication. When it does not, the disciplined conclusion is uncertainty, not a more elaborate marketing chart.
Count the choices that could have changed the winner
Consider a research grid with five lookback lengths, four entry thresholds and three exit rules. Even before changing markets or dates, that is 60 combinations. Choosing the start date after seeing results, dropping an inconvenient asset, or switching from Sharpe ratio to maximum drawdown adds further discretion. The trial log should record these decisions even when they were made in a notebook rather than by an optimizer. Otherwise the reported number of “strategies tested” understates the search that produced the final claim.
For a prespecified family of 20 valid hypothesis tests, a Bonferroni adjustment tests each at 0.05 / 20 = 0.0025 to bound the chance of any false rejection by 5%. This bound does not require the tests to be independent, although it can be conservative when they overlap. NIST’s discussion of simultaneous comparisons explains the underlying inequality. The adjustment cannot repair biased data, invalid individual tests or a hidden sequence of additional experiments. Knowing the arithmetic is not the same as knowing the real family of hypotheses.
There is also a difference between protecting against any false positive and controlling the expected proportion of false positives among discoveries. Those are different statistical objectives; neither establishes that a selected strategy is economically viable. Choose an approach for the research question rather than trying several corrections and reporting whichever permits the most exciting result. Repeatedly checking whether a result has become significant creates another selection problem unless the sequential procedure accounts for those repeated looks.
A reproducible selection test must rerun the selection itself. If a simulation or resampling exercise generates alternative datasets but evaluates only the already-selected winner, it misses the advantage gained by searching across many candidates. Instead, each replicate should apply the declared candidate search, selection rule and subsequent evaluation under a defensible data model. That is harder than drawing a confidence band around one attractive equity curve, but it addresses the procedure that actually generated the claim.