Multiclass SVM + Ineffective X Validation, Time Series Prediction

Question

I've recently run into an interesting and rather odd problem with cross validating a multiclass SVM that I can't figure out. Basically, I have a timeseries to predict and have created a dataset of what I believe are relevant predictors (also timeseries). I've done some preprocessing (reduced the example size of the dataset using a priori knowledge). I'm left with about 3000 training examples and a test set of 700 with about 650 features. When I do stratified 5-fold X validation on my training set, my average training error is fairly low. However, when I test the final model on my test set, my error is very high. In addition, I've noticed that my training error increases dramatically when I run regular 5-fold (non-stratified) X validation, which preserve the order of the timeseries (e.g. trains on older examples to predict newer ones and vice versa, whereas stratified will have various examples from different times mixed together).

I started with stratified because my target data is unbalanced. Additionally, I'm using class weights due to difference in misclassification costs. Unfortunately, I have basically just selected some weights using a priori knowledge and not via X validation due to computational feasibility. Eventually I will try to use X validation.

Any thoughts on what could cause such a difference between the stratified and non-stratified 5 fold X validation? To me, I thought maybe certain training examples seem to predict certain test examples better depending on close in time they were. I was considering using sample weights or perhaps taking out my initial data preprocessing, though that will increase my computation time (30000 vs 3000 examples). Any suggestions and recommendations are most welcome. Let me know if any additional information on my problem will help.

Answer 1

You can't use standard cross-validation techniques on time series because the observations aren't independent. This may be the source of your problems. If you are randomly sampling to get your five folds for cross-validation, your CV statistics will be mistakenly low due to correlations across the folds. If your test set on which you calculate the final model is somehow more independent from those five folds (for example, maybe you split training/cv vs. test by time?) you will see poor performance on the test set.

For instructions on properly handling time series cross-validation, see Rob Hyndman's overview of cross-validation (scroll down for time series coverage) and his example of time series cross-validation using R. Rob describes using a rolling origin for time series cross-validation that seems like it might work for you.

Why are you seeing different results for your stratified vs. non-stratified cross-validations? In the unstratified case, it sounds like you are creating training and validation sets that are less correlated than in the stratified case, since you are splitting by time when doing unstratified cross-validation. This might lead to higher cross-validation error because of less correlation across the sets. In the stratified case, you say you have "various examples from different times mixed together." In this case, the training vs validation sets have more correlation, leading to lower training error, representative of overfitting rather than useful predictive power.