I just read a paper by Lopez de Prado and Lewis that describes how multiple testing increases the likelihood of a false positive discovery.
There is a crucial point that I do not understand in the paper:
The theorem assumes that there are k
samples from a variable (here: Sharpe Ratio) that follows a normal distribution. The paper claims (in my understanding) that this models a researcher testing a strategy "k-times" on the same data.
But then, what does the "true" Sharpe ratio measure? The ability of the researcher? (e.g. if "the" strategy comes in k different "flavours", i.e. parameter variations).
I would understand the claim if it meant that k independent samples are drawn from the same strategy (i.e. a moving average strategy with the same window length and k disjoint time periods). But the paper claims it models a researcher twisting his/her model k-times.
In the paper, they sample not returns, but sharpe ratios, so there are K sharpe ratios in each of S samples. So, each SR_k in each of the S samples represents the measured performance of a strategy.
What is the difference between, say SR_k,n
and SR_{k+1},n
?
They may be: (1) the same strategy tested over different time windows or (2) slightly different versions of the same strategy tested over the same time window. Intuitively, if it is (2), then Exhibit 1 in the paper explains very well, why one should not reuse the data if back-testing gave bad results. But if it is (1), then the result is rather trivial: You will always find a time window, where the strategy performs good “just by chance”. This would not be too much news then.
Can anyone help here?