0
$\begingroup$

it says in the Scikit learn documentation for cross validation: https://scikit-learn.org/stable/modules/cross_validation.html

When evaluating different settings (“hyperparameters”) for estimators, such as the C setting that must be manually set for an SVM, there is still a risk of overfitting on the test set because the parameters can be tweaked until the estimator performs optimally. This way, knowledge about the test set can “leak” into the model and evaluation metrics no longer report on generalization performance.

What does it mean that "parameters can be tweaked until estimator performs optimally"?

What scenarios are they describing?

I am thinking that you are only building a SVM using the training set. Sure, you can tweak this value C (which is the regularization parameter associated with the SVM optimization problem). But it would overfit on the training set, no matter how you tweak it, and not on the test set. So I am not understanding what leakage this portion is talking about.

Are they describing a scenario where someone is tuning C on the test set?

Can someone please explain? Thank you.

$\endgroup$
1
  • 1
    $\begingroup$ Yes, they are referring to tuning the parameter on the test set. $\endgroup$
    – Tim
    May 27 at 6:15
2
$\begingroup$

What it is saying is that if you optimize your parameters based on the test set (that you didn't train on) performance, you might be simply overfitting on your test set, not reflecting the real performance of the model in real life situation. That's why a validation set is used. The performance of your model should be approximately the same on both of these sets.

To avoid splitting the data into three part, the cross validation is introduced, allowing you to save some data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.