2
$\begingroup$

Do I need a validation set if I am using cross validation and grid search for parameter tuning? Similar to this question - what I understood is that it helps to prevent overfitting but it's not necessary. I've read dozens of other posts on stack exchange, but now I'm starting to see conflicting answers and I am very confused...

In train-validation-test, the validation set is used to tweak parameters. But in my training set, I am using 10 fold cross validation with grid search (simultaneously) for finding the parameters of my random forest model. I'm not going to change the parameters, because I trust GridSearch already found the best ones. I have performance metrics (sen, spe, PPV, NPV) and they seem good enough so I have no reason to suspect the model is bad. So what's the point of using a validation set other than "proper" science?

Note: My dataset is 600 samples, and split 70-30% (train-test for now). I will apply my model to the test set just once report my final performance metrics. I don't want a validation set since my dataset is already super tiny.

$\endgroup$

2 Answers 2

1
$\begingroup$

Your approach seems right. Usually, cross-validation replaces the need of a validation data set. So, the process is like this:

  1. Train/Test split

  2. Choose final model by cross-validation on the training data. This includes tuning. And retraining on the full training data.

  3. Evaluate on the test data to get an impression of true performance (and bias) of your mdoel.

Since your data is rather small, I'd reduce the size of the test data to e.g. 20%. Furthermore, since training is fast, I'd use repeated cross-validation to get more robust decisions.

$\endgroup$
1
$\begingroup$

There's two reasons why you may want a validation (or holdout test set or whatever you want to call it) that you do not look at, at all, while deciding on your final model and use to evaluate the final performance. If you are instead talking about a validation set for tuning hyperparameters (and have reserved a separate test set), then no, you typically do not need such a validation set in addition. For that purpose cross-validation is generally preferable (multiple appropriate splits are more stable/reliable for making decisions such as hyperparameter tuning than a single split).

Reasons for having a holdout test set:

Firstly, you tuned your modeling decisions (such as hyperparameters) based on the cross-validation (obviously, more of an issue when your dataset is smallish and less of an issue when it is huge; note: large/huge is bigger than many people think).

Secondly, your cross-validation may not resemble your actual prediction task (e.g. a classic case is if you are trying to predict a time series into the future, in that case 10-fold cross-validation can be extremely unreliable, while a holdout of the last month/year/whatever of your data can be a lot more meaningful). You may of course argue that that's more a question of doing cross-validation properly.

On a large enough dataset with a cross-validation that closely mimicks your real-world setting, experience e.g. in Kaggle competitions suggests that your cross-validation will often provide a really good estimate for the performance on a hold-out test set. On the other hand, there are examples that even with large datasets you eventually get overfitting to a validation/test set (see e.g. https://arxiv.org/abs/1902.10811). As you suggest it increases credibility/removes questions if you have a hold-out test set.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.