1
$\begingroup$

I have 3 kinds of models (Lineair regression, random forest regression and the gradient boosted forest regression). Normally I would apply CV to all 3 of them and use a validation set for that and use the best model to apply on the test set. However due to time constraints (it's for an assignment) and minimal computing power I'm not going to cross validate.

So my question is which approach should I take:

1) Divide set in training and test:

  • Train all 3 kinds of models (without CV) on the train
  • Test all 3 kinds of models on the test
  • Choose the best performing

2) Divide set in training, validation and test:

  • Train all 3 kinds of models (without CV) on the train
  • Test all 3 kinds of models on the validation
  • Choose the best performing one
  • Run it on the test set
$\endgroup$
2
$\begingroup$

Number 2 is the way to go.

Any "validation" that is used for deciding which model to keep/discard is really not a validation but part of the training procedure.

Thus, you need to validate that final model.

Approach 1 would give you a "best" model without you knowing how it performs: you'll only know that it likely performs worse than you think so far because of the optimistic bias caused by the selection step.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.