0
$\begingroup$

I am interested in training different algorithms on a data set and observing performance metrics. Currently, my approach is to train different algorithms on train data, and then evaluate performance on a test set. Then, I use a GridSearch with cross-validation to find the optimal hyper-parameters for the best-performing model and test again using those.

I am a unsure about cross-validation - Would this be the right approach or is there a way I could/should do cross-validation for all models?

Improve this question
$\endgroup$
1
  • $\begingroup$ The usual approach is to use cross-validation on the training data to choose the model, including selecting the hyperparameters, then train the chosen model on the full training set, and finally test this model on the test set (i.e. test once only). $\endgroup$
    – Henry
    Commented Apr 23, 2022 at 0:45

1 Answer 1

Reset to default
1
$\begingroup$

Would this be the right approach

Nope. You need to evaluate the process of selecting hyperparameters. This is done via nested cross validation.

I would encourage you to stop thinking of a single model. The real question we're trying to answer when we do cross validation is "how does my model creation process perform on unseen data".

To that effect, nested cross validation is a better approach. Here is how nested cross validation works.

  • Take your train data and split it into $k$ folds. Set 1 fold aside, let's call this set $X_v$. Take the remaining $k-1$ folds and combine them. Call this $X_t$.

  • Use $X_t$ to performa grid search cross validation. This means you're going to split $X_t$ into $k$ folds, leave out one, use the remainder to fit a model with the specified parameters in your grid, etc etc.

  • Once you've done grid search cross validation and selected a model, now use that model to predict on $X_v$. Note $X_v$ was not used in the selection procedure so there is no risk of data leakage, implicit or explicit.

If you're using sklearn, this very easily as shown here. Note that when you create an instance of GridSearchCV you create an estimator. So think of the process of selecting a model via grid search cross validation as a model in an of itself. This idea can help clarify when and how to use various validating schemes.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank for the answer. Would we run a grid search for each model? Similar to the answer here: stackoverflow.com/questions/44035275/nested-gridsearchcv. Your answer mention ensuring $X_v$ not being used in section procedure, but this is what's happening now already, the difference being that I am selecting only one model to use with $GridsearchCV$, and the selection is based on training and testing each model (DT, RF, NN) with default params. $\endgroup$
    – jinx
    Commented Apr 22, 2022 at 18:36
  • $\begingroup$ @jinx Yes, you would run a nested cross validation for each model. You choose the model which has the best nested cross validation score. Fit that model on your entire training sample (including hyperparameter selection), then predict on the test set to obtain an unbiased estimate of performance. $\endgroup$
    – Demetri Pananos
    Commented Apr 22, 2022 at 19:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.