2
$\begingroup$

The purpose of train, validate and test data splits addresses the issue of data leakage when tuning for the model's hyperparameters. Does Grid Search then eliminates the need for test set? Because grid search would already return the optimum hyperparamaters without users having to adjust the hyperparameters again. Hence, that makes the final left out test set redundant.

$\endgroup$
1

1 Answer 1

2
$\begingroup$

A good model selection question.

Because Grid Search would already return the optimum hyperparamaters without users having to adjust the hyperparameters again.

A grid type search for finding out best performing hyper-parameters qualifies as tuning, either values are selected on the grid or with an auxiliary algorithm, i.e., a Bayesian optimisation or random search methods.

that makes the final left out test set redundant.

No. Because we still need to select the "best" hyper-parameters on the grid and having a separate validation set for this purpose will reduce overtraining, i.e, in the sense that we shouldn't rely on the same dataset for both model parameters and hyper-parameters. Even for using grid-search, it is generally recommended to split training-validation sets and use the test for measuring the learning performance.

A great paper discussing the importance of keeping validation set could guide here us a bit better from Manchester & Liverpool teams:

On Splitting Training and Validation Set: A Comparative Study of Cross-Validation, Bootstrap and Systematic Sampling for Estimating the Generalization Performance of Supervised Learning Yun Xu & Royston Goodacre Journal of Analysis and Testing volume 2, pages 249–262 (2018) doi

Figure 1 from the paper: Model selection workflow

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.