i am training a SVM (RBF kernel) with a dataset of ~1500 samples (balanced) using fminsearch on the CV error for parameter optimization (C and s).

After i found the "best" parameters (local optima possible) i am retraining my model on the whole dataset to derive a "final" model.

Is this a wise thing to do? Would the final model be proned to overfitting?

I experience worse performance on unseen data which might be OK as the CV during my optimization approach produces a somewhat optimistic estimate on the test error.

I think this adresses a pretty general problem but i could not find proper reasoning yet... Would it be a reasonable alternative to use just one of the models from the best performing crossvalidation?

I assumed that once the parameters are fixed the model will not suffer from overfitting no matter how many more samples i use for training?




Yes, retraining on the whole dataset is generally the right thing to do. The optimal value of the regularisation parameter scales with the number of training examples, so increasing the regularisation parameter by the ratio of the size of the full training set and the size of the training set in cross-validation would be a reasonable thing to do (as otherwise the model may be slightly under-regularised), but the difference is usually so small as to make no practical difference.

Retraining on the full training set is likely to be the best approach if you want a single model, as the more training patterns are used, the lower the variance of the parameter estimates, and the more reliable the model is likely to be (provided the hyper-parameters are set to good values). In general, the more data, the better.

Sadly no model is ever completely free from over-fitting, whenever you optimise a criterion estimated on a finite sample of data, there is always a risk of over-fitting.

The important thing to remember is that CV is essentially a method of estimating the accuracy of a procedure for generating a model, rather than of the model itself.

Edit: An alternative would be to form an ensemble from the cross-validation models.

  • $\begingroup$ I am unclear what you mean by scaling the regularization parameter in this case. Are you suggesting that if model was originally trained using 90 cases per fold (e.g. 10 fold CV), yielding a lambda of 0.1, but then the final model, training on 100 cases would use a lambda of (100/90)*0.1? (I am referring to Friedmans regularization formulation here) $\endgroup$ – BGreene Nov 23 '12 at 15:04
  • $\begingroup$ Yes, something like that, the number of incorrectly classified patterns rises roughly linearly with the number of training patterns, and they are all likely to have non-zero values for the margin slacks. It is only a rough approximation and it generally makes little difference in practice, so I usually don't bother (for a while my implementation of the SVM penalised the mean of the margin slacks to normalise the value of C a bit from problem to problem, but again it doesn't make a great deal of difference. $\endgroup$ – Dikran Marsupial Nov 23 '12 at 15:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.