Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 2 down vote favorite
1

Just want to check that I am performing my cross validation procedures right. I'm using a non-linear svm. I do a five fold cross validation (5 splits of test/train on my original training data) and for each fold, run a grid search to find the optimal parameters for that train/test pair (e.g. best parameters where model is fitted on the training data then evaluated on the test data). After five iterations, for each parameter set (2 hyper-parameters), i have 5 fit measures. I just use the average of those 5 #s and find the parameter set with the best average. Does this sound correct? I'm pretty new to this so am not entirely aware of what other methods are there.

Additionally, was wondering a few more things: 1) Any way to make it faster? Cross validation + grid search is pretty computationally intensive. 2) Any other validation methods rather than k-fold (stratified I might add) cv? My data are timeseries so I was considering a moving window type validation, but wasn't sure. Any thoughts welcome.

I should add that I'm asking because my training fit (the average over the 5 cv folds) is still much better than my actual test data fit. I'm trying to figure out what might be causing this and the best way to reduce this difference. I realize increasing the # of folds may help though it also raises question 1) as well.

Thanks guys and thumbs up on the site. Great info.

share|improve this question
Just to clarify, when you find the optimal parameters, are you taking the test data into account? If so, you shouldn't be. You should be optimizing the fit on your training sample, then calculating the fit on the test sample w/o any further optimization. Also, your training fit will typically be better, often much better, than your test fit, simply because the parameters optimize the fit on the training sample but not on the test sample. (Part of) the point of the cross-validation is to estimate how good the out-of-sample fit really is. – jbowman Feb 10 '12 at 1:34
I am not taking the test data into account. I hear what you're saying about the test vs train fit, it's just that difference is so wide (sometimes 4-5x better on the train data) i thought i might be doing something wrong. – tomas Feb 10 '12 at 14:27
Wow, that's a lot, at least in my limited experience with SVM. What is your sample size? How many variables? – jbowman Feb 10 '12 at 14:48
My orig dataset is about 4700 examples. I use 700 for a test set so 4000 for my train. That 4000 is used in t5 fold x valid (so 800 test, 3200 train or so). I have a lot of features about 650. I'm wondering if I should use a few more folds for the validation because the optimal parameters for each fold can be quite different (so i think i have pretty large variance in the fit measure across folds). I haven't checked that specifically. I use python (scikits learn) so any speed up suggestions would also be great, esp if i increase the folds. – tomas Feb 10 '12 at 16:14

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.