3
$\begingroup$

Suppose we have some (large enough) labeled training set and use some exhaustive cross-validation technique (e.g. leave one out) for tuning hyperparameters of SVM (with some nonlinear kernel). Is it true, that if we replace one observation by some another, the best possible hyperparameters would not change much?

I use the following technique:

loss_function(parameters, data)
  mse = 0
  for i=1 to n:
    model = train(data[all_but_i], parameters)
    mse = mse + (predict(model, data[i]) - y[i])^2
  return mse

for which I minimize the loss_function given data as a function of parameters. Am I right treating this as LOO?

$\endgroup$

1 Answer 1

2
$\begingroup$

A model that does not change much if a small part of the training data is exchanged against a few other training cases is called stable, and I'd extend this also to hyperparameters.

You can check this type of stability or robustness e.g. by iterated/repeated cross-validation (though not by LOO).
Even the cross validation used for the tuning can give you valuable information about this if you do not average the results for the different surrogate models: do all CV splits agree on which hyperparameters are best? How much spread is there, are closeby hyperparameters producing closeby performance for all splits?

Sanity checks:

  • Look at the distribution of performance results for the same hyperparameters. How large a difference do you observe compared to this uncertainty? Does this allow you to select an optimum?

  • Does independent testing of the chosen hyperparameters yields very similar performance results to the inner performance results obtained during the grid search? If not, this is indication of overfitting which may mean that the optimization was driven by noise rather than by real performance differences. In this situation, the optimization is also not stable.


update: See here as a start about model stability measurements with iterated/repeated cross validation

If you need a paper: Beleites, C. & Salzer, R.: Assessing and improving the stability of chemometric models in small sample size situations, Anal Bioanal Chem, 390, 1261-1271 (2008).
DOI: 10.1007/s00216-007-1818-6

$\endgroup$
4
  • $\begingroup$ Can you please explain what does iterated/repeated cross-validation is? And I also cannot completely understand your answer: I ask about theoretical properties and as far as I understand your suggest to find out the answer empirically for a certain dataset? $\endgroup$
    – vladkkkkk
    Nov 4, 2015 at 15:24
  • $\begingroup$ Yes, I suggest to show empirically that your model is not affected. Theoretically, you'll always have improving stability with increasing number of cases, both for model parameters and model hyperparameters (there really isn't that much of a difference between estimating parameters and estimating hyperparameters - just that you use different heuristics how to estimate them). Thus, your question becomes meaningless if the enough data is defined as "enough cases to get stable hyperparameter estimates". $\endgroup$ Nov 4, 2015 at 16:48
  • $\begingroup$ So, what I have tested so far, an SVM for regression is very unstable, i.e. small changes in hyperparameters may lead to large enough changes in quality measure, e.g. MSE. $\endgroup$
    – vladkkkkk
    Nov 27, 2015 at 15:45
  • $\begingroup$ Well, how large is that change compare to the change in MSE when predicting the same sample with two SVMs trained on almost the same training set (exclude a few cases) with the same hyperparameters? And compared to the uncertainty of your MSE (variance of the mean)? And are the changes you observe for changing hyperparameters random (as unstable suggests) or do they form a trough with the minimum you're looking for? $\endgroup$ Nov 28, 2015 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.