1
$\begingroup$

I am new to machine learning and want to ask regarding a confusion I have. I have a data set which is labeled and I want to do supervised learning.

My question is related to cross-validation and validation set. I checked other SO question like this and a paper from academia.

I understand that I need training and test sets. I have labeled set of data, so I see that cross-validation may be an option for me to check my classifier accuracy (correct me if I'm wrong on the accuracy part). I want to ask if I plan to do the above, will validation set still have any significance ? I read that it is used to try different settings for the algorithms. If I do try different settings e.g. if I try Naive Bayes and do 10 fold cross-validation, and also want to try some settings for Naive Bayes, what is will be validation set in this situation or is a validation set even needed ?

$\endgroup$

2 Answers 2

1
$\begingroup$

I will borrow an example from Data Mining Techniques (by M Berry and G Linoff) which will hopefully help explain why you still need a true hold-out (validation) set which should never be touched during the model building process, and only to be used to evaluate the accuracy of the final model that you chose from the k-fold cross-validation process.

Imagine yourself back in the fifth grade. The class is taking a spelling test. Suppose that, at the end of the test period, the teacher asks you to estimate your own grade on the quiz by marking the words you got wrong. You will give yourself a very good grade, but your spelling will not improve.

If, at the beginning of the period, you thought there should be an 'e' at the end of "tomato", nothing will have happened to change your mind when you grade your paper. No new information have entered your system. You need cross validation.

Now, imagine that at the end of the test the teacher allows you to look at the papers of several neighbors before grading your own. If they all agree than "tomato" has no final 'e', you may decide to mark your own answer wrong. If the teacher gives the same quiz tomorrow, you will do better. But how much better? If you use the papers of the very same neighbors to evaluate your performance tomorrow, you may still be fooling yourself. If they all agree that "potatoes" has no more need of an 'e' than "tomato", and you have changed your own guess to agree with theirs, then you will overestimate your actual grade on the second quiz as well. That is why the the final validation set (i.e., the hold-out) should be different from the training/cross-validation set.

In summary, you can use cross-validation to select the best model (i.e., the best modeling technique, best parameters, and the optimal number of features). And to evaluate how your model will perform in future, you can check its accuracy on the validation set.

$\endgroup$
0
$\begingroup$

If you optimize your parameters (if any) via cross validation on the training set, then you can simply apply this model to the validation set to perform its accuracy. So in this case the need for a validation set might be for comparison with existing methods.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.