it says in the Scikit learn documentation for cross validation: https://scikit-learn.org/stable/modules/cross_validation.html
When evaluating different settings (“hyperparameters”) for estimators, such as the C setting that must be manually set for an SVM, there is still a risk of overfitting on the test set because the parameters can be tweaked until the estimator performs optimally. This way, knowledge about the test set can “leak” into the model and evaluation metrics no longer report on generalization performance.
What does it mean that "parameters can be tweaked until estimator performs optimally"?
What scenarios are they describing?
I am thinking that you are only building a SVM using the training set. Sure, you can tweak this value C (which is the regularization parameter associated with the SVM optimization problem). But it would overfit on the training set, no matter how you tweak it, and not on the test set. So I am not understanding what leakage this portion is talking about.
Are they describing a scenario where someone is tuning C on the test set?
Can someone please explain? Thank you.
