8
$\begingroup$

After reading some material, I found few options for defining train and test sets:

  1. Just splitting with no change.
  2. Accumulating/moving window of train set.
  3. Leave a relatively small (warming) period between test and train sets, and then use window again (including the warming period).

What should be the most accurate way for applying machine learning algorithms and parameters estimation?

Improve this question
$\endgroup$
4
  • $\begingroup$ The answer would depend on how much data you have, how many things are you trying to estimate, how correlated your data is, and how nonstationary the data might be. $\endgroup$
    – Memming
    Commented Mar 17, 2014 at 20:16
  • 1
    $\begingroup$ Obviously just splitting the data randomly as you normally would in supervised ML is a bad idea for time series. I've used essentially the same approach described at the end of this blog post successfully in the past, but with a step size larger than 1 to save computing time. The paper linked at the bottom was also helpful. $\endgroup$
    – alto
    Commented Mar 18, 2014 at 2:17
  • 1
    $\begingroup$ @Memming, you may consider an entire daily data of an asset (say 20 years). The data is highly correlated and up/down trend prediction. Thanks! $\endgroup$
    – Eitan
    Commented Mar 18, 2014 at 15:15
  • $\begingroup$ Thanks alto. Actually, thats exctly what i did but wasn't sure about it. Can the same approach be used for feature selection as well? $\endgroup$
    – Eitan
    Commented Mar 18, 2014 at 15:18

2 Answers 2

Reset to default
3
$\begingroup$

If you're still looking for insight regarding financial time series & machine learning, you might want to check out this article from the Journal of Economic Perspectives, which gives a great overview of various ML methods pertaining to Economics/Finance.

Essentially, the main problem you have is that most traditional machine learning techniques deal with cross-sectional data "where independently distributed data is a plausible assumption" (quoted from said article). However, since with Financial Time Series you, by and large, can't make that assumption, you're better off taking a totally different approach than the 'ole Training/Test set split-'em-up. Your best bet--as mentioned in that article (Seriously, it's really good)--may be to read up on Bayesian Structural Time Series (BFTS) (briefly mentioned in that article that you should be reading by now and described in more detail here and, well, I don't have the reps for a third link...).

Now, if you're just looking to do some run-of-the-mill Time Series estimation you can settle for the choose-the-model-with-the-lowest-out-of-sample-RMSE approach. However, that may cause you to forfeit all your "Machine Learning" name-dropping privileges. Just a warning...

Good luck!

Improve this answer
$\endgroup$
2
  • $\begingroup$ Hi Steve. I'm already using the idea mentioned in the link in @Alto's above answer. It's quite simple and intuitive (and works!). I will read the article, but i guess studing a completely new method should take a while. Thanks! $\endgroup$
    – Eitan
    Commented Jul 2, 2014 at 7:47
  • $\begingroup$ Hey, I'm just glad you still care (after all, your original post was a couple months ago...). Definitely check out that first article--it's by Hal Varian, who is now the chief economist at Google (plus, JEP articles tend to be much more readable than your typical Economics fare). $\endgroup$
    – Steve S
    Commented Jul 2, 2014 at 8:32
0
$\begingroup$

Generally cross-validation is one of the methods to evaluate a model by splitting data into train and test data sets. Leave-one-out cross-validation splits the dataset, say n datapoints as (n-1) for train data and test on nth datapoint. this process is repeated until each data point serves as a test datapoint. This ensures fairness in splitting the training data and rigorous evaluation of the model.

Improve this answer
$\endgroup$
1
  • 10
    $\begingroup$ If you blindly apply normal CV methods when working with highly correlated data, like things with temporal/spatial characteristics, you're going to have a bad time (and likely be highly overoptimistic about true model performance). $\endgroup$
    – alto
    Commented Mar 18, 2014 at 2:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.