17
$\begingroup$

I've been reading about k-fold validation, and I want to make sure I understand how it works.

I know that for the holdout method, the data is split into three sets, and the test set is only used at the very end to assess the performance of the model, while the validation set is used for tuning hyperparameters, etc.

In the k-fold method, do we still hold out a test set for the very end, and only use the remaining data for training and hyperparameter tuning, i.e. we split the remaining data into k folds, and then use the average accuracy after training with each fold (or whatever performance metric we choose to tune our hyperparameters)? Or do we not use a separate test set at all, and simply split the entire dataset into k folds (if this is the case, I assume that we just consider the average accuracy on the k folds to be our final accuracy)?

|cite|improve this question
$\endgroup$
  • 3
    $\begingroup$ It depends on what you want to do. If you want a generalized performance estimate, then yes, the model should make tests on data it has never seen before. But that doesn't mean it has to be a single holdout iteration, you can use resampling to achieve the same goal. $\endgroup$ – Firebug Jul 27 '16 at 19:22
  • 2
    $\begingroup$ ... which means that whenever you use "validation" results for hyperparameter optimization/model tuning, you need another stage of validation that is independent of that tuning. For both stages you can use e.g. either cross validation or hold out (or out-of-bootstrap or ...). CV + CV is called nested CV, hold out + hold out leads to the 3 set setup you mentioned $\endgroup$ – cbeleites Jul 28 '16 at 7:33
9
$\begingroup$

In the K-Fold method, do we still hold out a test set for the very end, and only use the remaining data for training and hyperparameter tuning (ie. we split the remaining data into k folds, and then use the average accuracy after training with each fold (or whatever performance metric we choose) to tune our hyperparameters)?

Yes. As a rule, the test set should never be used to change your model (e.g., its hyperparameters).

However, cross-validation can sometimes be used for purposes other than hyperparameter tuning, e.g. determining to what extent the train/test split impacts the results.

|cite|improve this answer
$\endgroup$
  • 6
    $\begingroup$ +1 but you might want to mention nested cross-validation as an alternative to the cross-validation + test set. $\endgroup$ – amoeba Jul 27 '16 at 19:13
  • 1
    $\begingroup$ "can sometimes be used for purposes other than hyperparameter tuning". For example, you can use cross validation for validation purposes (= testing unknown cases to measure generalization error). $\endgroup$ – cbeleites Jul 28 '16 at 7:31
3
$\begingroup$

Generally, yes. Basically you we are talking about the bias-variance tradeoff. If you use data to build up your model (training and validation data) and you iterate over different hyperparameters and you try to maximize an averaged performence metric your model might not be as good as indicated.

However, especially in small datasets the additional split might lead to an even smaller training set and result in a bad model.

|cite|improve this answer
$\endgroup$
  • 2
    $\begingroup$ This is not a real answer to the question. $\endgroup$ – Michael Chernick Feb 13 '17 at 15:08
  • $\begingroup$ Can you expand this so that it adds something to the accepted answer and the fairly detailed comments? $\endgroup$ – mdewey Feb 13 '17 at 15:57
1
$\begingroup$

Ideally, validation (for model selection) and final test should not be mixed. However, if your k value is high, or it is leave-one-out, using test result to guide your model selection is less harmful. In this scenario, if you are writing an academic paper, do not do it (unless you bother to explain)-- meaning always have a separate test set. If you are building a practical project, it is OK to do so.

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.