# Re-building a cross-validated SVM

Suppose we are cross-validating parameters of a Gaussian (radial) SVM on $$k$$ training observations. The parameters are the cost parameter $$C$$, and the deviation parameter $$\gamma$$.

Then, $$4k$$ more training observations arrive. If we train the SVM on the whole training set with the previously validated values $$C,\gamma$$ this will not be a good practice for obvious reasons. However, if we cross validate $$C,\gamma$$ on the training set on $$5k$$ observations, this will be too expensive.

Is there any way in between these two extreme practices, e.g., using directly parameters that are cross validated on the first $$k$$ observations versus cross-validating the whole $$5k$$ observations training set?

I would say, since we now have a lot more observations than before, we can instead assume the new $$4k$$ observations are the validation set. However, I am looking for an answer which is directly from the SVM practice. Maybe there is a clever way.

• Without having read the papers @usεr11852 linked (and fully expecting that this will be part of what they do): with more training data at hand now, you can considerably restrict the cost; gamma search space for the new data set since you expect the optimal solution to be of same or higher complexity - you'll probably not want to evaluate lower complexity models. Unfortunately, the high complexity ones are the computationally costly ones... (This assumes that you do not have systematic changes, i.e. drift in your data generation process) – cbeleites unhappy with SX May 21 at 21:33
• @cbeleitesunhappywithSX Thanks for your comment. What do you mean by 'of higher complexity'? – independentvariable May 21 at 22:38

• thanks so much! So can we say that maybe we can first validate the trained parameters on the new $4k$ set. If we see that the error is still low, then the two datasets are alike. However, if the error is different, and we expect more and more data set coming in, then we should do incremental learning? I thought incremental learning is usually for the models where hyperparameters are fixed (e.g., in SVM, fix support vectors throw the rest away, bring the new data, and go on). – independentvariable May 21 at 21:32
• Cool. Yes, I have seen this latest paper (+1). Based on the question's text ($C$ and $\gamma$) I thought you had a fixed kernel type too. Clearly not all hyper-parameters are equally easy to retrain, if we cannot assume some kind of "path" (e.g. ala Hastie et al. (2004) we do not have any significant advantage when retraining. – usεr11852 May 22 at 0:14