ENH: lib: Add deflated_sharpe_ratio() to judge optimize() results - #1388
Open
ipezygj wants to merge 1 commit into
Open
ENH: lib: Add deflated_sharpe_ratio() to judge optimize() results#1388ipezygj wants to merge 1 commit into
ipezygj wants to merge 1 commit into
Conversation
The best run of Backtest.optimize() is the maximum over all tried parameter combinations, so its Sharpe ratio is inflated by multiple testing: the expected best Sharpe of N skill-less trials grows with N. Add lib.deflated_sharpe_ratio(stats, trial_sharpe_ratios), computing the probability the winning Sharpe ratio exceeds zero after correcting for the number and dispersion of trials actually made (Bailey & Lopez de Prado 2014, https://doi.org/10.3905/jpm.2014.40.5.094). Uses only stdlib statistics.NormalDist — no new dependencies. The periodic-returns resampling is extracted from compute_stats() into _stats.periodic_returns() and reused, not duplicated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
lib.deflated_sharpe_ratio(stats, trial_sharpe_ratios)— the probability (0–1) that the best run returned byBacktest.optimize()has a Sharpe ratio genuinely greater than zero, after correcting for the multiple testing the optimization itself performs.Why
optimize()returns the maximum over all tried parameter combinations. The expected best Sharpe of N skill-less trials is already well above zero and grows with N, so the winning run's Sharpe is inflated by selection — the more combinations tried, the more the "best" result reflects luck rather than edge. The library currently reports the winner's Sharpe with no way to ask whether it clears that bar. This is the standard correction: Bailey & López de Prado (2014), The Deflated Sharpe Ratio (doi:10.3905/jpm.2014.40.5.094).The trial count and the trial Sharpe dispersion are taken from the search itself (the heatmap), not from an assumed default —
optimize()already produces exactly the data the estimator needs:Concrete example with the test
SmaCrossonGOOG(28 combinations): the winner's Sharpe looks significant as a single test (PSR ≈ 0.98) but does not clear the hurdle its own 28-trial search sets by chance (DSR ≈ 0.18). That flip is the information the number adds.Implementation notes
statistics.NormalDist; the rest is numpy/pandas already in use.compute_stats()into_stats.periodic_returns()and reused, so the winner's returns are computed by the exact same code path as the reported Sharpe — no duplicated freq logic.'Sharpe Ratio'(i.e. optimization maximized something else, so the values wouldn't be trial Sharpes).flake8clean; fullpython -m backtesting.testsuite passes.