One path among many
It is no secret that markets are in different phases over time. In addition to the classic, calm bull market, there are volatile sideways phases, turbulent crashes and gruelling bear markets. Each of these phases ends at some point and passes over into a new regime. The characteristics of the individual phases also change over time, so that no market is identical to the previous version of itself. The individual players adapt and learn, and as a result, the markets evolve over time.
The challenge of developing permanently profitable algorithms under these constantly changing conditions is enormous. In his paper "Tactical Investment Algorithms", Marcos López de Prado explains the crucial problem: "We do not know the true process that generated the past data time series. All we have is one of many paths that could have developed. Therefore, simple back tests based on this one path alone are on shaky legs.
Better Backtests
In view of these challenges, are there any trading strategies that "always" work? If de Prado has his way, the chances of that are slim - and even if they did exist, they probably wouldn't be outstanding.
However, this all-weather assumption is implicit in a test method that is particularly common in theory and practice: walk-forward analysis. At first glance, it may seem more mature than simple backtesting due to the inclusion of out-of-sample periods, but ultimately the two basic problems remain the same: The assumption that future observations are based on the same data generating process (DGP) as the past ones. And the assumption that the one observed path is actually representative of this process.
As de Prado writes in his paper, advanced test methods such as bootstrapping can be used to solve the problematic path dependency by resampling past values. But even here, according to de Prado, the problem of limited data history remains, which is not necessarily representative for future development.
Monte Carlo, the clear favorite
In his paper, de Prado favours the Monte Carlo method. Based on a given estimated DGP, this method simulates a large number of data sets whose statistical properties correspond to those of the DGP and which are overlaid with random noise. Unlike bootstrap resampling, for example, where the simulation is based on actually observed market data, the Monte Carlo method uses artificial data. As a classic example, the paper describes a regime switching model in which samples can be taken from different DGPs. A special algorithm is used to continuously estimate the probability that the DGP will switch to a new regime. A model parameterized in this way can then be optimized to match the statistical properties of the data actually observed. Based on these objectives, a large number of artificial data sets can be replicated, allowing for a much more comprehensive analysis than would be possible by resampling a limited and potentially unrepresentative data history.
In addition to the statistical properties, sound theoretical considerations can also contribute to a deeper understanding of the probable DGP. De Prado cites as an example two dynamic variables between which, according to accepted theories, a long-term equilibrium exists (cointegration). Empirical studies can be used to determine the approximate range of values that these variables can assume. Based on these inputs, it is possible to simulate many millions of years of artificial data in which the variables realize all possible values within the estimated bandwidth.
Is there a catch?
Normally, the fact that algorithms are tested with the Monte Carlo approach on the basis of artificial data is often criticized - this might not be meaningful for future data realized by DGP. But de Prado does not accept this criticism for two reasons. First, estimating a suitable DGP is not necessarily more difficult than a market forecast: anyone who is convinced that statistical methods enable successful investment results should also be convinced that these methods can be used to identify a DGP. Secondly, even with walk forward and resampling analyses, these observations are unlikely to occur exactly as simulated, so the synthetic paths generated with Monte Carlo are no less likely.
