Machine learning (ML) in markets is the application of data-driven algorithms — penalized linear models, tree ensembles, neural networks, and reinforcement-learning agents — to financial decision problems: predicting returns, ranking a cross-section of stocks, sizing or gating a signal, executing an order, or allocating a portfolio. Its appeal is that markets are awash in high-dimensional, non-linear data and ML excels at finding interactions a human cannot specify in advance. Its defining tension is equally simple: financial data has an extraordinarily low signal-to-noise ratio, is non-stationary (the data-generating process drifts and regimes change), and is cheap to backtest against millions of times — so the very flexibility that makes ML powerful is exactly what makes it overfit and "discover" edges that do not exist. The whole discipline, correctly practiced, is therefore less about exotic architectures than about honest validation and managing the gap between in-sample fit and out-of-sample reality.

What this section covers

This is a section node. It maps the sub-topics of applying ML to markets and points to the child docs for depth — it does not repeat their detail. The four children together trace a pipeline (raw data → features → model → validation) plus the alternative action-learning paradigm:

• Feature engineering — transforming raw prices, volumes, fundamentals, order-book and text data into the numerical inputs a model consumes. Widely regarded as the highest-leverage, most domain-specific step (which features you feed often matters more than which model). Covers price/return transforms, technical indicators as features, volatility/liquidity/cross-sectional features, and finance-specific techniques like fractional differentiation (stationarity without destroying memory) — and warns that lookahead/leakage is the number-one failure mode.

• Supervised learning for returns — training a model on labeled feature→future-return examples, framed as regression (predict the return) or classification (predict a label, e.g. via López de Prado's triple-barrier method). Covers the canonical cross-sectional long-short application, the meta-labeling pattern (a secondary model sizes/gates a primary signal), and the field's benchmark evidence (Gu, Kelly & Xiu 2020).

• Overfitting & regularization — the bias-variance tradeoff, regularization techniques (L1/L2, dropout, early stopping, tree constraints), and the larger problem of selection/backtest overfitting: purged k-fold and combinatorial purged cross-validation, the Deflated Sharpe Ratio, and the Probability of Backtest Overfitting. This is the methodology node that protects every other one.

• Reinforcement learning — the distinct paradigm that learns actions and sequences (buy/sell/hold, order size, limit-order placement) directly against a risk-adjusted reward, rather than predicting a label. Covers MDP/POMDP framing, value-based vs policy-gradient methods, the differential Sharpe reward (Moody & Saffell), and the use cases with the most credible track record (execution, cost-aware allocation) versus the unproven one (alpha generation).

The core tension, stated once

Three properties of markets make ML here unlike ML elsewhere, and every child doc is an instance of dealing with them:

1. Low signal-to-noise. The legitimately extractable signal is small. Gu, Kelly & Xiu (2020) — the field's most-cited benchmark, ~30,000 U.S. stocks 1957–2016 — found their best neural-net forecasts produced monthly out-of-sample R² only in the ~0.33–0.40% range (their NN3 specification reports 0.40%), yet that tiny number compounded across a wide cross-section into an economically large long-short Sharpe (the paper reports an annualized 1.35 value-weighted / 2.45 equal-weighted decile-spread Sharpe for the NN forecasts, vs. 0.61 / 0.83 for OLS). The lesson: aim for a small, stable edge over breadth, not high accuracy on a noisy target.

2. Non-stationarity. A model perfectly validated on one period can break in the next because the regime changed, not because it overfit. No amount of regularization fixes this; it is why a separate regime layer matters.

3. Backtest overfitting is the default, not the exception. Bailey, Borwein, López de Prado & Zhu showed it is a mathematical near-certainty: with only ~5 years of daily data, trying roughly 45 strategy variations is enough to find one with an in-sample Sharpe ≥ 1.0 purely by chance even when the true Sharpe is zero. Hence the corpus-wide insistence on purged/embargoed, leakage-free, multiple-testing-aware validation.

When it matters vs. when it doesn't

ML earns its keep where the cross-section is wide (many assets average out idiosyncratic noise), the horizon is slower (monthly cross-sectional ranking is far more tractable than tick-level direction), the objective is well-defined and gradable (execution shortfall, cost-aware allocation), and validation is rigorous. It is weakest — often worse than a simple rule — when applied to a single asset's price level (near-random-walk), pushed for directional accuracy on noisy short-horizon data, or evaluated only in-sample. Across the field the durable signals are unglamorous and ML-agnostic — variants of momentum, liquidity, and volatility (Gu/Kelly/Xiu); ML's contribution is modeling their non-linear interactions, not conjuring new predictors. Exotic mined features rarely survive live.

Standing & evidence (honest landscape)

ML moved from fringe to mainstream in academic asset pricing after Gu/Kelly/Xiu (2020) and the Kelly–Xiu survey (NBER w31502, 2023), and is standard at quant funds and bank execution desks. But the honest landscape is contested on three fronts the children detail: net-of-cost profitability (much paper alpha lives in small, illiquid names and erodes after transaction and shorting costs), post-publication decay (documented; magnitude debated), and false discoveries (López de Prado's central thesis that most published financial-ML results are artifacts of improper validation). Reinforcement-learning results are similar — strong in backtest, guarded in reality, with no broadly replicated evidence of a durable directional edge after realistic costs. The reconciliation the corpus adopts: the edge is real but small, fragile, and easy to fake, and the validation-first posture is the responsible default.

Sources

• Gu, S., Kelly, B., Xiu, D. (2020). "Empirical Asset Pricing via Machine Learning." Review of Financial Studies 33(5): 2223–2273. PDF: https://dachxiu.chicagobooth.edu/download/ML.pdf — benchmark out-of-sample R², dominant predictors (momentum/liquidity/volatility), long-short Sharpe.
• Kelly, B., Xiu, D. (2023). "Financial Machine Learning." NBER WP 31502 — https://www.nber.org/system/files/working_papers/w31502/w31502.pdf (field survey: ML as a prediction/risk-premium task; promising directions and caveats).
• López de Prado, M. (2018). Advances in Financial Machine Learning, Wiley — triple-barrier labeling, meta-labeling, fractional differentiation, purged/embargoed CV, feature importance.
• López de Prado, M. (2018). "The 10 Reasons Most Machine Learning Funds Fail," Journal of Portfolio Management 44(6): 120–133 — false discoveries, multiple-testing deflation.
• Bailey, Borwein, López de Prado, Zhu — "Pseudo-Mathematics and Financial Charlatanism" (Notices of the AMS, 2014) / "The Probability of Backtest Overfitting" — the ~45-trials-on-5-years result and the False Strategy Theorem. https://davidhbailey.com/dhbpapers/backtest-prob.pdf
• Child docs in this section (supervised learning, feature engineering, overfitting & regularization, reinforcement learning) for full mechanics, evidence and per-topic sources.

Confidence: medium. This is a section-overview synthesis; precise figures (R² ~0.33–0.40%/month, Sharpe ~1.35, ~45-variation overfitting threshold) are the widely-reported headline values and vary by specification — treat as representative, not canonical. Net-of-cost ML profitability and the magnitude of post-publication decay are genuinely contested.