2
$\begingroup$

I just read a paper by Lopez de Prado and Lewis that describes how multiple testing increases the likelihood of a false positive discovery.

There is a crucial point that I do not understand in the paper:

The theorem assumes that there are k samples from a variable (here: Sharpe Ratio) that follows a normal distribution. The paper claims (in my understanding) that this models a researcher testing a strategy "k-times" on the same data.

But then, what does the "true" Sharpe ratio measure? The ability of the researcher? (e.g. if "the" strategy comes in k different "flavours", i.e. parameter variations).

I would understand the claim if it meant that k independent samples are drawn from the same strategy (i.e. a moving average strategy with the same window length and k disjoint time periods). But the paper claims it models a researcher twisting his/her model k-times.

In the paper, they sample not returns, but sharpe ratios, so there are K sharpe ratios in each of S samples. So, each SR_k in each of the S samples represents the measured performance of a strategy.

What is the difference between, say SR_k,n and SR_{k+1},n?

They may be: (1) the same strategy tested over different time windows or (2) slightly different versions of the same strategy tested over the same time window. Intuitively, if it is (2), then Exhibit 1 in the paper explains very well, why one should not reuse the data if back-testing gave bad results. But if it is (1), then the result is rather trivial: You will always find a time window, where the strategy performs good “just by chance”. This would not be too much news then.

Can anyone help here?

$\endgroup$
1
  • $\begingroup$ Good Question! This paper is also referenced in this blog with an accompanying very nice plot. However, I am not able to track down the source of that plot. $\endgroup$ – not2qubit Jun 19 '18 at 19:36
2
$\begingroup$

As I understand the claim, researchers have access to $k$ strategies (or assets, funds, whatever), and have backtested them (or observed their returns), each over a period of, say, $T$ days, and their returns are independent. In that case, the sample Sharpe ratios of those $k$ strategies are independent, and are asymptotically normal with standard error given by the standard formula (which dates back to Johnson & Welch), or to the Mertens' form (which includes population skew and kurtosis). I believe in Lopez de Prado's retelling, the most basic standard error is used, which is $1/\sqrt{T}$, under the null that all of the strategies have zero expected return. (nb if the returns were normally distributed, each sample Sharpe ratio would take a $t$ distribution. I am not aware of an asymptotic $t$ approximation for the case of non-Gaussian returns.)

As you can imagine, the assumption of independence is highly questionable. I examine the effect of simple correlation to a single factor in this blog post. The silver lining is that correlation seems to make the test conservative, rather than anti-conservative, so an upper bound on type I errors is probably maintained (although at considerable loss of power). I plan a followup blog post to examine the 'Markowitz approximation' which has higher power for this problem. (A preview can be found in section 3.1 of this vignette for SharpeR.)

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.