In what situations would cross validations scores be inaccurate? I'm trying to fit a SVM model on times series stock return data, predicting a buy, hold, or sell signal of the stock. I'm using 10-fold cross validation (using the R package caret), and I'm getting very high precision and recall scores. However, when I test the model on a new sample the performance is much worse (i.e. 40% precision in new sample vs 80% precision in 10-fold cv). I thought that the cv process would prevent overfitting, but the results suggest otherwise. Can anyone provide some thought as to what might be going on here and some ways to deal with this? Thank you!!
 A: Time series data from stock market is not independent observation. Hence using cross-validation does not remove all the associated information due to correlation with other observations.
You can obtain cross-validation stats for time-series data by:


*

*Fit the model to the data $y_1,\dots,y_t$ and let $\hat{y}_{t+1}$ denote the forecast of the next observation. Then compute the error ($e_{t+1}^*=y_{t+1}-\hat{y}_{t+1}$) for the forecast observation.

*Repeat step 1 for $t=m,\dots,n-1$ where m is the minimum number of observations needed for fitting the model. (How will you know $m$? You'll have to plot the learning curve and determine the minimum number of sample that are needed for your model to converge.)

*Compute the MSE from $e_{m+1}^*,\dots,e_{n}^*.$

A: You seem to have misunderstood what CV does. CV is an evaluation technique to assess the performance of a given classifier, when the sample size is not large enough to set aside a test set. CV helps avoid overfitting only indirectly by letting you assess the performance of certain parameter configurations of a given classier (such as SVM in your case), and let you choose one with the highest metric of your choice (e.g. accuracy, precision etc). Inner CV is typically employed for that purpose.
Given the situation you presented, i would say the performance of your classification system (combination of features, feature selection and classifier) is only 40%, which is what represents the true generalization performance. The 80% precision you noticed is only average metric from across the various folds, if caret behaves similar to typical toolboxes.
Perhaps, if you provide the exact code, we can offer more specific hints. 
