I've recently run into an interesting and rather odd problem with cross validating a multiclass SVM that I can't figure out. Basically, I have a timeseries to predict and have created a dataset of what I believe are relevant predictors (also timeseries). I've done some preprocessing (reduced the example size of the dataset using a priori knowledge). I'm left with about 3000 training examples and a test set of 700 with about 650 features. When I do stratified 5-fold X validation on my training set, my average training error is fairly low. However, when I test the final model on my test set, my error is very high. In addition, I've noticed that my training error increases dramatically when I run regular 5-fold (non-stratified) X validation, which preserve the order of the timeseries (e.g. trains on older examples to predict newer ones and vice versa, whereas stratified will have various examples from different times mixed together).
I started with stratified because my target data is unbalanced. Additionally, I'm using class weights due to difference in misclassification costs. Unfortunately, I have basically just selected some weights using a priori knowledge and not via X validation due to computational feasibility. Eventually I will try to use X validation.
Any thoughts on what could cause such a difference between the stratified and non-stratified 5 fold X validation? To me, I thought maybe certain training examples seem to predict certain test examples better depending on close in time they were. I was considering using sample weights or perhaps taking out my initial data preprocessing, though that will increase my computation time (30000 vs 3000 examples). Any suggestions and recommendations are most welcome. Let me know if any additional information on my problem will help.