2
$\begingroup$

I just started to learn data mining. I have learned that cross validation can be used to compare models (e.g. svm vs logistic, etc) and to do tuning (like regularization parameter) .

I have a dataset. I split it into training and testing set and then used training to do cross validation to compare different models and then tuning the "winner model" on the same cross validation to get the most out of it.

Will training and tuning on the same data lead to overfit?

(In coursera machine learning forum, I have seen an instructor say that each set of data can only be used for one purpose. If you use cv to adjust the regularization parameter then you cannot use cv to select the polynomial degree.)

I have seen lots of articles saying that cv can be used to compare models and do tuning, but I do not know. If I want to do both, what procedures should I follow? If I have several models in my mind and I use cv to compare those models, then I find out that svm is good, but how should I tune it further? Use the same cv set? or use another validation set to estimate?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Will training and tuning on the same data lead to overfit?

Yes. There are two chief ways around this: use a holdout validation set (making sure to hold it out before doing any of the tuning CV), or use nested CV (that is, inside each iteration of the CV you use to estimate predictive accuracy, use another round of CV on the training partition to tune the model, then use the tuned values to train it on the whole training partition).

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you very much, so just use inner loop to compare different models and use outer loop to tune the parameter (like regularization parameter), is my understanding correct? thank you ! $\endgroup$
    – flashing sweep
    Commented Jan 29, 2017 at 4:02
  • $\begingroup$ Also may I ask a irrelevant question, how many times am I allowed to apply model to the test set? My instructor told us that we can only apply it once or it will be part of training set, but I have seen so many examples that they applied it to test set and then go back to correct models, is that legit?Thank you ! $\endgroup$
    – flashing sweep
    Commented Jan 29, 2017 at 5:11
  • $\begingroup$ "so just use inner loop to compare different models and use outer loop to tune the parameter (like regularization parameter)" — No, do it the other way around. You want to compare the tuned models, not the untuned models, right? $\endgroup$
    – Kodiologist
    Commented Jan 29, 2017 at 16:28
  • $\begingroup$ "how many times am I allowed to apply model to the test set?" — The devil's in the details, but your instructor is essentially right. It is okay to see that testing performance is lower than you'd like and to add some more models for that reason. By contrast, if you make fine adjustments of a model to optimize test-set performance, such as by setting tuning performance, then the danger arises of overfitting to your test set, so you'd want yet more data to examine the performance of the final model without any bias from overfitting. See also stats.stackexchange.com/q/235591 $\endgroup$
    – Kodiologist
    Commented Jan 29, 2017 at 16:28
  • $\begingroup$ @WendyHuang Sure, don't forget to accept my answer if it's satisfactory. You can do that by clicking the check mark under the voting arrows. $\endgroup$
    – Kodiologist
    Commented Jan 29, 2017 at 23:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.