Apologies for my naivety if the answer to the question is simple, stats is an area I am not comfortable in and am looking to improve.

My problem is within the frame of finance. Simply put, say I have a base strategy which gives a certain time series of returns when backtested. This is my baseline.

I then apply a signal as a hopeful improvement on this baseline return series. That results in a different return series.

There are many questions that could be asked at this point, and finance provides some basic metrics that are commonly used to evaluate the difference between these two time series (sharpe, max drawdown, return to drawdown, ect.). These do not utilize statistics though, so I do not know for sure if, say, a sharpe of 1.3 for the base strategy is different from a sharpe of 1.4 for the alternative strategy.

What I want to know is are these two time series different? Secondly, is one better? The question "what is better?" needs to be answered. For now let's just make it simple, and say that a strategy is better than the base if it's compounded return over the series is greater than the compounded return of the base strategy (it makes more money). Alternatively this could be a higher risk adjusted return, or any number of the other finance metrics.

This question makes sense to me outside of the time series space (so just comparing two samples, are the means different? Is one mean greater than another? simple), but I'm getting a bit hung up on the path dependency of the strategies. Then of course there are the assumptions (if any) on the underlying distribution of returns, a concept I've found is often thrown by the wayside in finance (just assume normal) for ease of computation.

If anyone has any insights into this, any literature that they may suggest reading, or that sort of thing it'd be much appreciated. I just need to be pointed in the right direction, my googling has not yielded great (or maybe more accurately, accessible) results.


If you consider the returns to be stationary, a very simple way to compare two return time series is using a cross-sectional view of the data (i.e. histogram). This actually what the Sharpe ratio does. (I didn't understand what exactly you meant when you said those metrics "didn't use statistics". Do you mean p-value, confidence regions, etc? Because the Sharpe ratio is pretty much the mean of the returns divided by the standard deviation, which certainly are "statistics" of the data).

If you subscribe to a frequentist view, you can use a Student's t-test to compare the two distributions. If you are Bayesian, you can use Kruschke's BEST approach, which is a little more involved but the result is more interpretable.

For example, a Bayesian analysis would give you the probability that the mean of the returns of one asset is greater than the other, while a frequentist would give you the probability of rejecting the null hypothesis (which could be that the two samples have the same mean) if the null hypothesis is actually true.

If you go for a cross-sectional approach, you have to be careful on your assumptions about the data. First, you must confirm that the returns are stationary. Second, you must choose a distribution to model the returns. People in finance usually use Gaussians for that, but it underestimates extreme events ("black swans"), hence it underestimates risk.

Lastly, defining what is better in finance is an open issue. The problem arises when dealing with risk. Some people don't even believe that it's possible to model risk analytically (such as Nassim Taleb). Most people use standard deviation as risk, but it's a very superficial approach. A metric I like is conditional value-at-risk (CVaR) because it doesn't ignore tail events (e.g. low probability of huge losses) and is convex, which makes it easier to embed in an optimization procedure. Of course, you will be completely dependent on the way you model and estimate parameters from the data, but most metrics will suffer from the same problem.

  • $\begingroup$ Thanks for the response Pedro, yes that was a mis-use of verbiage on my part with "didn't use statistics". I was referring to significance testing, confidence regions, ect as you said. My first question would be using the student's t-test makes the assumption that my sample (the returns) follow a t-distribution, I don't think that's a valid assumption but is it a "close enough" assumption? Second, is Box-Jenkins a way to go about the cross sectional approach? My understanding of that methodology is that it transforms the series to stationary, then estimates (basically). $\endgroup$ – DMT Feb 2 '15 at 18:49
  • $\begingroup$ Then would you continue by comparing the two estimations of the model (one for each strategy) for differences? $\endgroup$ – DMT Feb 2 '15 at 18:50
  • $\begingroup$ @DMT actually the Student's t-test I cited compares the means of two Gaussian distributions. The Student's t distribution arises as the solution to the problem (it's used to calculate the p-value). Using a Student's t distribution to model the returns is a different beast, and is a better approximation than a Gaussian, but the frequentist statistics are not so easy to come by. Luckily, the Bayesian BEST considers the samples to come from the Student's t distribution, so if you wanna model like that, you can use it instead of the t-test. $\endgroup$ – Pedro Tabacof Feb 2 '15 at 18:58
  • $\begingroup$ When you have the time series of the prices and you take the returns, you are getting a (more) stationary series. The Box-Jenkins also checks for seasonality, but I don't think this is a issue here. You can use Box-Jenkins to check if your assumptions make sense, but hopefully you won't need to transform the data anymore after taking the returns. Sorry, I didn't understand your second comment. $\endgroup$ – Pedro Tabacof Feb 2 '15 at 19:02
  • $\begingroup$ Sorry I wasn't clear. I think I'm just latched on to the notion that I need to hypothesis test, so my second comment was really asking "To get an idea of whether or not the two strategies are different, could you fit arima models to both of them, and then test the coefficients in the same way you would compare two linear regressions" $\endgroup$ – DMT Feb 2 '15 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.