Should I use a validation set when not cross validating?

Question

I have 3 kinds of models (Lineair regression, random forest regression and the gradient boosted forest regression). Normally I would apply CV to all 3 of them and use a validation set for that and use the best model to apply on the test set. However due to time constraints (it's for an assignment) and minimal computing power I'm not going to cross validate.

So my question is which approach should I take:

1) Divide set in training and test:

• Train all 3 kinds of models (without CV) on the train
• Test all 3 kinds of models on the test
• Choose the best performing

2) Divide set in training, validation and test:

• Train all 3 kinds of models (without CV) on the train
• Test all 3 kinds of models on the validation
• Choose the best performing one
• Run it on the test set

Answer 1

Number 2 is the way to go.

Any "validation" that is used for deciding which model to keep/discard is really not a validation but part of the training procedure.

Thus, you need to validate that final model.

Approach 1 would give you a "best" model without you knowing how it performs: you'll only know that it likely performs worse than you think so far because of the optimistic bias caused by the selection step.