I would like to test a proprietary method of financial time series prediction that says, in effect, that on a certain day/week in the future, to an accuracy of +/- 2 to 3 days/weeks, a financial time series will make a relative high or low on this predicted date, and furthermore, within a defined period in the future there will only be a certain number of such relative highs and lows.

What statistical tests exist that can ascertain whether such a predictive methodology is non-random and actually has predictive ability in as much as it accurately calls the highs and lows? For the purpose of such test(s) the years 2007 to 2011 inclusive can be considered unseen, out of sample data.

One idea I have is to plot the data and count how far the actual highs and lows are from their predicted dates of occurrence and devise an error metric and then randomly permuting the data to obtain new time series with random highs and lows, applying the same error metric and then comparing the real error metric with the distribution of the random error metrics.


Your suggestion is to devise some "error metric" (a test statistic) and compare the predictor on real and resampled data.

You could also use the same idea, on real data, to compare your predictor with many random predictors: this would give you an idea of the distribution of your error metric, under the (null) hypothesis that the predictor is random, and the quality of your predictor can then be expressed as a "bootstrap p-value".

  • $\begingroup$ A very interesting twist on my idea and, what is more, your idea will be much easier computationally - it won't require permutation of the whole data set and nor will it require an algorithm to identify the new highs and lows of these randomly permuted "alternative futures." $\endgroup$ – babelproofreader Jan 26 '12 at 11:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.