Sample size for actual vs backtested performance

I have 10 years of backtested simulated performance of some trading strategy (using historical prices), and N months of actual trading performance. I want to compare the two. How big does N have to be to get a statistically significant comparison? I'm comparing returns and Sharpe ratios.

-
maybe you can think about moving that question to quant.stackexchange if you don't get an answer here. – Zarbouzou Jun 30 '11 at 16:39
OK. I'll also post the question there. For future reference, how do you transfer questions from one forum to another? – Nestor Jun 30 '11 at 18:17
Better to flag your question for mods' attention so that they can migrate it. – chl Jun 30 '11 at 20:29
Also posted here: quant.stackexchange.com/questions/1375/… – Nestor Jun 30 '11 at 21:56

A slightly simpler formulation is as follows: suppose you believe that your trading strategy has a Sharpe ratio of $\psi$, 'annualized' to the time units of your mark frequency (monthly in your case, evidently). Then to perform a 1-sided, 1-sample t-test for the null hypothesis that the expected return of your strategy is zero, you should set $$N = \frac{2.7}{\psi^2}$$ in order to have a power of 0.5 and a type I rate of 0.05 (the 'magical' value). Note this is just a modification of Lehr's rule (as described by Van Belle).

There are a large number of caveats here:

1. this only holds for reasonably non-skewed distributions of returns.
2. using relative returns (percents) instead of log returns will create a geometric bias.
3. having a small number of samples biases the estimate of Sharpe.

There is probably a similar formula for the two sample t-test to compare mean returns, or a test to compare Sharpe ratios, but I don't know them (yet).

-
So if I have a monthly Sharpe of 0.75 = Avg(Monthly Returns)/Stdev(Monthly Returns), my N comes to 2.7/0.75^2 = 4.8 months? – Nestor Jun 30 '11 at 21:34
If your monthly Sharpe is 0.75, then your annualized Sharpe is 2.6, which is rather high! The above rule should probably be amended to state that you should probably have at least 20 samples. (I assume you have access to daily marks. In that case, the length of time remains the same, but the number of samples changes). So barring the small number problem, yes, in 4.8 months, you should have a reasonably powerful test under that value of the alternative. Most likely what will happen is you will fail to reject the null and be somewhat disappointed. – shabbychef Jun 30 '11 at 21:38
Thanks shabbychef. 4.8 seems consistent with the minimum track record a lot of investors ask (i.e. 6 months). I've posted a follow up question to this: stats.stackexchange.com/questions/12521/… if you care to chip in. – Nestor Jun 30 '11 at 21:49