BACKTESTING
HOW TO AVOID FALSE DISCOVERIES
We invested over 2000 hours in developing one of the most advanced backtesting engines currently available on the market.
A robust backtesting engine is critical for two different reasons:
- It helps us avoid false discoveries while developing trading systems.
- It provides us means to identify abnormal performance deterioration soon.
Step 1: Early Stage
We always base our trading systems on economic theory and continue our analyses only if the in-sample performance on the first half of the available data looks promising.
After that, we optimize the parameters and report their sensitivity analysis to check their robustness.
Subsequently, we perform the in-sample permutation tests proposed by Masters (2019) to detect overfitting.
We continue with the subsequent analyses only when these early results look interesting.
Step 2: Walk-Forward and Nested Walk-Forward
In this stage, for each trading system, we perform a classic walk-forward analysis and a nested walk-farward analysis to consider selection bias. At this stage, it is critical to ensure that there is no future leakage into our analysis (Masters, 2018) and test for the soundness of our trading factory using permutations (Masters, 2020).
This part is concluded by estimating the Probabilities of Backtesting Overfitting (Bailey et al. 2015) in the newest version proposed by Masters (2020).
Step 3: Simple and Combinatorial Cross-Validation
At first, a purged cross-validation with proper embargo is performed. Subsequently, the Combinatorial Cross-Validation (De Prado, 2018) is employed to obtain a more robust result. The average Sharpe ratio coming from this procedure is tested against the null hypothesis that is equal to or lower than the target Sharpe ratio.
Finally, the Combinatorially Simmetric Cross-Validation (Bailey, 2015) is employed to test for the presence of overfitting.
Only when results from Steps 2 and 3 are reassuring will we continue with the following parts of the backtesting engine.
Step 4: Simulations
At first, we estimate the trading system performance using Monte Carlo Simulations and Bootstrapping. These metrics, while incomplete, provide a first assessment of the probability of ruin.
After that, many stochastic market models specification is fitted to market data.
Subsequently, simulations are performed for each stochastic model fitted previously (De Prado, 2018).
Finally, the performance of each trading system is analyzed for the different simulation sets.
Step 5: Risk Analysis
At first, the strategy’s performance is analyzed across different market conditions and the different stages of the business cycle.
Then, the worst negative results detected are analyzed in light of the trading system’s economic rationale.
Subsequently, a confidence interval for the Mean trade return is estimated using the bias-corrected and accelerated bootstrap method (Masters 2018).
Finally, the Lower bound for the Value at Risk and drawdown of the studied trading system are computed following a two stages bootstrapping algorithm (Masters 2018).
Step 6: Detailed Performance Analysis
The last stage of our backtesting engine consists of a detailed Performance Analysis of the strategy returns: per trade, per day, and month.
We consider many performance metrics, including probabilistic Sharpe Ratio, Deflated Sharpe ratio, Sortino Ratio, and Information Ratio, …
We consider the average VaR per trade and per day, the maximum drawdown, and the time under-water for each trading system.
