Backtesting is the simulation of a trading rule against historical data to estimate how it would have performed; validation is the separate, harder discipline of deciding whether that estimate can be believed as a forecast of live behavior. The two are not the same thing, and conflating them is the original sin of quantitative trading. A backtest is trivially easy to make look spectacular — the same machinery that fits genuine market structure also fits the noise of one particular price history, and a researcher who tries enough variants is statistically guaranteed to find an impressive curve by luck alone. This section is therefore not about how to run a backtest (any platform does that) but about the much more valuable question of when a backtest is lying. The core tension running through every child node is that a backtest's optimism and its trustworthiness move in opposite directions: every parameter tuned, every variant tried, and every peek at the test set makes the historical equity curve prettier and the out-of-sample expectation worse. Validation is the set of defensive disciplines that fight that drift.

When this domain matters (and when it doesn't)

These methods are essential whenever a number derived from history — a win rate, a Sharpe ratio, an expectancy, a "this setup works 68% of the time" claim — is going to drive a real capital or sizing decision. They matter most for parameterized, systematic strategies with optimization in the loop, for any data-mined "anomaly," and for universes with high attrition (small caps, micro caps, distressed names, leveraged or short-volatility books) where the deleted losers are precisely the tail risk. They matter least for a hand-reasoned thesis that was never optimized against the data it's evaluated on, or for liquid large-cap, low-turnover universes with few delistings. The unifying rule the section encodes: the more degrees of freedom you exercised against the data, the more aggressively you must discount the result — and if you cannot say how many variants you tried or name the as-of date of every input, assume the number is inflated.

Map of the sub-topics

This branch decomposes the validity problem into four pillars — two modeling failures and two data failures, plus the methodologies that defend against them.

• Overfitting & Curve-Fitting (001) — the central modeling failure: a strategy tuned so tightly to one history that it learns noise, not structure. This is the conceptual hub of the section. It covers the mechanics (excess degrees of freedom, parameter optimization, multiple-testing selection bias, iterative researcher bias) and the formal corrective toolkit from the Bailey / López de Prado program: the False Strategy Theorem, Minimum Backtest Length, the Deflated Sharpe Ratio, and the Probability of Backtest Overfitting (PBO/CSCV). If you read only one child node, read this one.

• Out-of-Sample Testing (004) — the foundational defense: split history, develop on the in-sample block, judge only on a held-out block the optimizer never saw. Covers split conventions (70/30, 80/20 — conventions, not theory), why shuffled k-fold cross-validation is invalid for time series, and the central caveat that a holdout is only honest the first time it is touched — iterated, peeked-at holdouts silently become training data.

• Walk-Forward Analysis (002) — the professionalized form of OOS testing (Robert Pardo, 1992/2008): re-optimize on a rolling in-sample window, trade the next unseen window, roll forward, and stitch the OOS segments into one deployment-realistic equity curve. Covers anchored vs. rolling windows, Walk-Forward Efficiency, parameter stability as a robustness signal, and the second-order ("meta") overfitting trap. Often called the "gold standard" — a label the node treats as industry consensus, not proven fact, noting López de Prado's argument for Combinatorial Purged Cross-Validation over single-path walk-forward.

• Look-Ahead & Survivorship Bias (003) — the two data-provenance failures that no amount of out-of-sample testing can fix, because they corrupt both windows equally. Look-ahead bias leaks future information (reporting lag, restatements, today's index membership) into past decisions; survivorship bias tests only the names that survived, deleting the dead. Covers the point-in-time (PIT) data and delisting-inclusive-universe defenses, and measured magnitudes (e.g. Bessembinder's finding that only ~42.6% of US common stocks over 1926–2016 had lifetime buy-and-hold returns exceeding one-month T-bills).

Together these are the four checks behind the single question this section exists to answer: can this historical number be trusted forward?

Standing & evidence

That backtests systematically over-promise is among the most settled findings in empirical finance, not folklore. The industry frames it canonically as the "Seven Sins of Quantitative Investing" (Luo et al., Deutsche Bank, 2014), which include survivorship, look-ahead, storytelling, data mining, ignoring transaction costs/slippage, and outliers. The academic critique is sharper still: Bailey, Borwein, López de Prado and Zhu (2014) showed that with enough trials any Sharpe ratio is achievable by chance, and Harvey, Liu & Zhu (2016) argued that most of the hundreds of published asset-pricing "anomalies" fail a proper multiple-testing correction — finance's version of the replication crisis. What remains genuinely contested is how much the corrections help: the Deflated Sharpe Ratio and PBO require estimating the number of effective independent trials, which is hard because variants are correlated, and even a clean OOS result can be implicitly mined through reuse. The honest consensus the children converge on: these defenses lower the danger materially but eliminate it for none — and markets' non-stationarity means even a genuinely un-overfit edge can still decay.

Sources

• Bailey, Borwein, López de Prado & Q.J. Zhu (2014), Pseudo-Mathematics and Financial Charlatanism, Notices of the AMS — backtest-overfitting framework and Minimum Backtest Length (MinBTL). davidhbailey.com/dhbpapers/backtest-pseudo.pdf
• Bailey & López de Prado (2018), The False Strategy Theorem (SSRN 3221798) — formal statement that the expected max Sharpe across N trials is right-unbounded.
• Bailey & López de Prado (2014), The Deflated Sharpe Ratio — papers.ssrn.com/sol3/papers.cfm?abstract_id=2460551
• Luo et al. (Deutsche Bank, 2014), Seven Sins of Quantitative Investing — survivorship, look-ahead, data mining, transaction costs; summarized at portfoliooptimizationbook.com/book/8.2-seven-sins.html
• Harvey, Liu & Zhu (2016), …and the Cross-Section of Expected Returns — multiple-testing critique of the factor zoo.
• Robert Pardo, The Evaluation and Optimization of Trading Strategies (Wiley, 2008) — walk-forward analysis, efficiency.
• Marcos López de Prado, Advances in Financial Machine Learning (2018) — purging, embargoing, CPCV, deflated Sharpe.
• Bessembinder (2018), Do Stocks Outperform Treasury Bills?, JFE — survivorship-relevant base rates.
• Child nodes (this branch): Overfitting & Curve-Fitting, Out-of-Sample Testing, Walk-Forward Analysis, Look-Ahead & Survivorship Bias.

Dispute flagged: the "gold standard" label for walk-forward and the corrective metrics (DSR/PBO/MinBTL) are influential but rest on estimating the effective number of independent trials, which is approximate; they reduce but do not remove overfitting risk. The direction of all four biases (always over-optimistic) is uncontested; their magnitude is asset-class- and era-dependent.