# In what situations would cross validations scores be inaccurate?

I'm trying to fit a SVM model on times series stock return data, predicting a buy, hold, or sell signal of the stock. I'm using 10-fold cross validation (using the R package caret), and I'm getting very high precision and recall scores. However, when I test the model on a new sample the performance is much worse (i.e. 40% precision in new sample vs 80% precision in 10-fold cv). I thought that the cv process would prevent overfitting, but the results suggest otherwise. Can anyone provide some thought as to what might be going on here and some ways to deal with this? Thank you!!

• What did you do CV on? held-out data or training data? Dec 30 '15 at 6:11
• @VihariPiratla I used cv on about half of my training data and then tested the resulting model on the other half (I guess this is the held-out data?)
– xyy
Dec 30 '15 at 6:27
• What all SVM params are you changing by CV? Also, what are the value intervals for each parameter in which you are considering the optimal value. What is the size of your training data? My guess is that your model is too complex or/and your training data is too small, which could be the reason(s) for over-fitting. Dec 30 '15 at 9:44
• How did you choose your features? Did you also use CV for that? Dec 30 '15 at 10:02
• @JimBoy I did not use CV for feature selection. Currently my features are values of various technical indicators (such as MACD) for a period of time preceding the prediction period. Any thoughts on how I might incorporate CV into that?
– xyy
Dec 31 '15 at 19:52

Time series data from stock market is not independent observation. Hence using cross-validation does not remove all the associated information due to correlation with other observations. You can obtain cross-validation stats for time-series data by:

1. Fit the model to the data $y_1,\dots,y_t$ and let $\hat{y}_{t+1}$ denote the forecast of the next observation. Then compute the error ($e_{t+1}^*=y_{t+1}-\hat{y}_{t+1}$) for the forecast observation.
2. Repeat step 1 for $t=m,\dots,n-1$ where m is the minimum number of observations needed for fitting the model. (How will you know $m$? You'll have to plot the learning curve and determine the minimum number of sample that are needed for your model to converge.)
3. Compute the MSE from $e_{m+1}^*,\dots,e_{n}^*.$
• Thanks for the answer! Is this essentially the same as LOOCV? Why does this work better than the 10-fold CV I'm currently using? Also I'm new to stats, would you mind explaining a bit more about the learning curve process or point me to some article on that? thank you!!
– xyy
Dec 31 '15 at 20:24
• In LOOCV (Leave on out Cross Validation) you are only leaving one sample for test and rest all samples are used in training. This method is susceptible to high variance (over-fitting) hence not very useful unless you don't have enough training data. If you have enough data just use K-fold CV. For time-series data though, each training data is not an independent observation, so you'll have to select the K-fold in "chunks". Just selecting any one observation will have to same high variance problem that LOOCV has. Jan 1 '16 at 17:40
• Learning curve is the method to plot the error against the number of training samples. You'll then plot the training and cross validation errors. There will come a point when adding more training sample stops reducing the training and cross validation error further. This is the minimum number of training data you need for this model. Jan 1 '16 at 17:50

You seem to have misunderstood what CV does. CV is an evaluation technique to assess the performance of a given classifier, when the sample size is not large enough to set aside a test set. CV helps avoid overfitting only indirectly by letting you assess the performance of certain parameter configurations of a given classier (such as SVM in your case), and let you choose one with the highest metric of your choice (e.g. accuracy, precision etc). Inner CV is typically employed for that purpose.

Given the situation you presented, i would say the performance of your classification system (combination of features, feature selection and classifier) is only 40%, which is what represents the true generalization performance. The 80% precision you noticed is only average metric from across the various folds, if caret behaves similar to typical toolboxes.

Perhaps, if you provide the exact code, we can offer more specific hints.

• thank you so much for your answer! I'm still confused by the cv score is so much higher than a new data set, since I assume the cv score is the average of the performance on the held-out fold for each iteration?
– xyy
Dec 30 '15 at 6:29
• This answer seems a little confusing to me; while not the only use, the vast majority of the time the motivation for the use of CV is to prevent overfitting. Dec 30 '15 at 16:37
• Yes - vast majority of the use cases for CV are in parameter tuning. CV does help you prevent overfitting - albeit indirectly - by helping you in your selection for the model (parameter tuning) with highest generalizability, which is typically the one with least overfitting. Dec 30 '15 at 18:08
• Can you explain the heavy emphasis on "indirectly"? The problem of overfitting is that as you allow a model to get more complex, after awhile further reducing the in-sample error actually increases out of sample error (so using in-sample metrics, like MSE on your whole dataset, never catch overfitting). CV aims to estimate out of sample error. As such, CV is a tool which (to me) seems to directly address the issue of overfitting. Dec 31 '15 at 6:21
• There you go, you answered your own question. CV helps you estimate out of sample error, which is its primary purpose. It is not a regularization technique, not a direct one like Tikhonov or Least Squares, is what I meant. Dec 31 '15 at 13:24