9
$\begingroup$

What is the difference between training, test, and holdout sets?

I know these concepts, just want to ensure that I have understood correctly.

Training set is something that we have as of now. We will remove subset from it and removed subset will be called holdout set.

We will build models using remaining data (what remains after removing holdout set) and the holdout set is used to finalized estimates of tuning parameters (step 1)

Then we will build a final model on the entire Training set (including holdout set). Tuning parameters values are same as that we got from step 1.

Test data is something that we get in future. We don't know their Y/dependent variable value and we predict it using our model.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
11
$\begingroup$

Well, Hastie, Tibshirani, and Friedman, in their seminal The Elements of Statistical Learning (page 222), say to break the data into three sections:

  1. Training (50%)
  2. Validation (25%)
  3. Testing (25%)

Where the model is built on the training set, the prediction errors are calculated using the validation set, and the test set is used to assess the generalization error of the final model. This test set should be locked away until the model calibration process is finished to prevent underestimation of the true model error.

Hastie, T.; Tibshirani, R. & Friedman, J. The Elements of Statistical Learning: Data Mining, Inference and Prediction Springer Science+Business Media, Inc., 2009

Improve this answer
$\endgroup$
3
  • $\begingroup$ which data is used to calculate tuining parameters? $\endgroup$
    – user2543622
    Commented Aug 29, 2013 at 18:19
  • $\begingroup$ If I understand them properly, the validation set is used to calibrate the tuning parameters. $\endgroup$
    – Avraham
    Commented Aug 29, 2013 at 18:28
  • 2
    $\begingroup$ Consider hyperparameters (such as the lamda used for regularization, the sigma used in the kernel function of a SVM, or the number of hidden layers and neurons per layer in a neural network) as separate from the base parameters of the algorithm. You set parameters during training, tune hyperparameters in validation, and avoid any tuning based on test. A key goal of finding the right hyperparameters is to find the right balance between overfitting and underfitting (variance and bias, respectively). $\endgroup$
    – Paul Jackson
    Commented Sep 7, 2016 at 18:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.