Just want to check that I am performing my cross validation procedures right. I'm using a non-linear svm. I do a five fold cross validation (5 splits of test/train on my original training data) and for each fold, run a grid search to find the optimal parameters for that train/test pair (e.g. best parameters where model is fitted on the training data then evaluated on the test data). After five iterations, for each parameter set (2 hyper-parameters), i have 5 fit measures. I just use the average of those 5 #s and find the parameter set with the best average. Does this sound correct? I'm pretty new to this so am not entirely aware of what other methods are there.
Additionally, was wondering a few more things: 1) Any way to make it faster? Cross validation + grid search is pretty computationally intensive. 2) Any other validation methods rather than k-fold (stratified I might add) cv? My data are timeseries so I was considering a moving window type validation, but wasn't sure. Any thoughts welcome.
I should add that I'm asking because my training fit (the average over the 5 cv folds) is still much better than my actual test data fit. I'm trying to figure out what might be causing this and the best way to reduce this difference. I realize increasing the # of folds may help though it also raises question 1) as well.
Thanks guys and thumbs up on the site. Great info.