3
$\begingroup$

I've recently run into an interesting and rather odd problem with cross validating a multiclass SVM that I can't figure out. Basically, I have a timeseries to predict and have created a dataset of what I believe are relevant predictors (also timeseries). I've done some preprocessing (reduced the example size of the dataset using a priori knowledge). I'm left with about 3000 training examples and a test set of 700 with about 650 features. When I do stratified 5-fold X validation on my training set, my average training error is fairly low. However, when I test the final model on my test set, my error is very high. In addition, I've noticed that my training error increases dramatically when I run regular 5-fold (non-stratified) X validation, which preserve the order of the timeseries (e.g. trains on older examples to predict newer ones and vice versa, whereas stratified will have various examples from different times mixed together).

I started with stratified because my target data is unbalanced. Additionally, I'm using class weights due to difference in misclassification costs. Unfortunately, I have basically just selected some weights using a priori knowledge and not via X validation due to computational feasibility. Eventually I will try to use X validation.

Any thoughts on what could cause such a difference between the stratified and non-stratified 5 fold X validation? To me, I thought maybe certain training examples seem to predict certain test examples better depending on close in time they were. I was considering using sample weights or perhaps taking out my initial data preprocessing, though that will increase my computation time (30000 vs 3000 examples). Any suggestions and recommendations are most welcome. Let me know if any additional information on my problem will help.

$\endgroup$
5
$\begingroup$

You can't use standard cross-validation techniques on time series because the observations aren't independent. This may be the source of your problems. If you are randomly sampling to get your five folds for cross-validation, your CV statistics will be mistakenly low due to correlations across the folds. If your test set on which you calculate the final model is somehow more independent from those five folds (for example, maybe you split training/cv vs. test by time?) you will see poor performance on the test set.

For instructions on properly handling time series cross-validation, see Rob Hyndman's overview of cross-validation (scroll down for time series coverage) and his example of time series cross-validation using R. Rob describes using a rolling origin for time series cross-validation that seems like it might work for you.

Why are you seeing different results for your stratified vs. non-stratified cross-validations? In the unstratified case, it sounds like you are creating training and validation sets that are less correlated than in the stratified case, since you are splitting by time when doing unstratified cross-validation. This might lead to higher cross-validation error because of less correlation across the sets. In the stratified case, you say you have "various examples from different times mixed together." In this case, the training vs validation sets have more correlation, leading to lower training error, representative of overfitting rather than useful predictive power.

$\endgroup$
  • $\begingroup$ Thanks Anne I'll take a look at your references. I had thought about this myself and just realized that this was what the issue is. I'm fairly sure my data has significant autocorrelation. Thanks again. $\endgroup$ – tomas Feb 11 '12 at 17:31
  • $\begingroup$ You're welcome. I'm dealing with correlated data too (time plus clustering) and have been working through similar cross-validation issues. $\endgroup$ – Anne Z. Feb 11 '12 at 17:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.