What does "multiple backtesting" mean in the "False Strategy Theorem" of de Prado and Lewis?

Question

I just read a paper by Lopez de Prado and Lewis that describes how multiple testing increases the likelihood of a false positive discovery.

There is a crucial point that I do not understand in the paper:

The theorem assumes that there are k samples from a variable (here: Sharpe Ratio) that follows a normal distribution. The paper claims (in my understanding) that this models a researcher testing a strategy "k-times" on the same data.

But then, what does the "true" Sharpe ratio measure? The ability of the researcher? (e.g. if "the" strategy comes in k different "flavours", i.e. parameter variations).

I would understand the claim if it meant that k independent samples are drawn from the same strategy (i.e. a moving average strategy with the same window length and k disjoint time periods). But the paper claims it models a researcher twisting his/her model k-times.

In the paper, they sample not returns, but sharpe ratios, so there are K sharpe ratios in each of S samples. So, each SR_k in each of the S samples represents the measured performance of a strategy.

What is the difference between, say SR_k,n and SR_{k+1},n?

They may be: (1) the same strategy tested over different time windows or (2) slightly different versions of the same strategy tested over the same time window. Intuitively, if it is (2), then Exhibit 1 in the paper explains very well, why one should not reuse the data if back-testing gave bad results. But if it is (1), then the result is rather trivial: You will always find a time window, where the strategy performs good “just by chance”. This would not be too much news then.

Can anyone help here?

Answer 1

As I understand the claim, researchers have access to $k$ strategies (or assets, funds, whatever), and have backtested them (or observed their returns), each over a period of, say, $T$ days, and their returns are independent. In that case, the sample Sharpe ratios of those $k$ strategies are independent, and are asymptotically normal with standard error given by the standard formula (which dates back to Johnson & Welch), or to the Mertens' form (which includes population skew and kurtosis). I believe in Lopez de Prado's retelling, the most basic standard error is used, which is $1/\sqrt{T}$, under the null that all of the strategies have zero expected return. (nb if the returns were normally distributed, each sample Sharpe ratio would take a $t$ distribution. I am not aware of an asymptotic $t$ approximation for the case of non-Gaussian returns.)

As you can imagine, the assumption of independence is highly questionable. I examine the effect of simple correlation to a single factor in this blog post. The silver lining is that correlation seems to make the test conservative, rather than anti-conservative, so an upper bound on type I errors is probably maintained (although at considerable loss of power). I plan a followup blog post to examine the 'Markowitz approximation' which has higher power for this problem. (A preview can be found in section 3.1 of this vignette for SharpeR.)