2
$\begingroup$

Suppose we are cross-validating parameters of a Gaussian (radial) SVM on $k$ training observations. The parameters are the cost parameter $C$, and the deviation parameter $\gamma$.

Then, $4k$ more training observations arrive. If we train the SVM on the whole training set with the previously validated values $C,\gamma$ this will not be a good practice for obvious reasons. However, if we cross validate $C,\gamma$ on the training set on $5k$ observations, this will be too expensive.

Is there any way in between these two extreme practices, e.g., using directly parameters that are cross validated on the first $k$ observations versus cross-validating the whole $5k$ observations training set?

I would say, since we now have a lot more observations than before, we can instead assume the new $4k$ observations are the validation set. However, I am looking for an answer which is directly from the SVM practice. Maybe there is a clever way.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Without having read the papers @usεr11852 linked (and fully expecting that this will be part of what they do): with more training data at hand now, you can considerably restrict the cost; gamma search space for the new data set since you expect the optimal solution to be of same or higher complexity - you'll probably not want to evaluate lower complexity models. Unfortunately, the high complexity ones are the computationally costly ones... (This assumes that you do not have systematic changes, i.e. drift in your data generation process) $\endgroup$
    – cbeleites unhappy with SX
    May 21, 2020 at 21:33
  • $\begingroup$ @cbeleitesunhappywithSX Thanks for your comment. What do you mean by 'of higher complexity'? $\endgroup$
    – independentvariable
    May 21, 2020 at 22:38

1 Answer 1

Reset to default
1
$\begingroup$

I think we can treat this as an instance of online (or incremental) machine learning. That is, given this new data use assume that the training data arrives in a continuous manner and we need to tune our hyper-parameters on the fly. In a way, we are doing a "warm start" in our problem.

There are quite a few papers on the matter particular for SVM:

Please note that this is definitely not confined to SVM. Most algorithms that are trained in an iterative manner have been presented within an online/incremental learning framework.

Improve this answer
$\endgroup$
8
  • $\begingroup$ thanks so much! So can we say that maybe we can first validate the trained parameters on the new $4k$ set. If we see that the error is still low, then the two datasets are alike. However, if the error is different, and we expect more and more data set coming in, then we should do incremental learning? I thought incremental learning is usually for the models where hyperparameters are fixed (e.g., in SVM, fix support vectors throw the rest away, bring the new data, and go on). $\endgroup$
    – independentvariable
    May 21, 2020 at 21:32
  • 1
    $\begingroup$ @independentvariable: hyperparameters are not really all that different from other parameters. The main difference is that they are parameters the developer of the algorithm couldn't left for you to fix, because they did not have an automated and generally applicable way to fix them. $\endgroup$
    – cbeleites unhappy with SX
    May 21, 2020 at 21:35
  • 1
    $\begingroup$ @independentvariable I am happy I could help. We can definitely try what you said but realistically, with more data, we will like to get potentially a more generalisable and/or robust model. As cbeleites said, hyperparameters are not necessarily fixed. $\endgroup$
    – usεr11852
    May 21, 2020 at 21:37
  • 1
    $\begingroup$ However, this one addresses also updating the hyper parameters: ieeexplore.ieee.org/abstract/document/… $\endgroup$
    – independentvariable
    May 21, 2020 at 23:43
  • 1
    $\begingroup$ Cool. Yes, I have seen this latest paper (+1). Based on the question's text ($C$ and $\gamma$) I thought you had a fixed kernel type too. Clearly not all hyper-parameters are equally easy to retrain, if we cannot assume some kind of "path" (e.g. ala Hastie et al. (2004) we do not have any significant advantage when retraining. $\endgroup$
    – usεr11852
    May 22, 2020 at 0:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.