1
$\begingroup$

I have a question regarding the CV in GridSearchCV.

To test my model should I split my data into 3: training, validation, test? For easy understanding let's say my data is split into training with 60% of the data, validation data 20%, and test data is the remaining 20%. Should I use my training data to train the model to tune my parameters with the validation data and then test it on test data?

However, if I use GridSearchCV, I already use Cross Validation (=5 for example) which already splits the data in training. Does this mean that by using GridSearchCV I only need to split data into training and test? And does this help with overfitting?

Should I divide my data into 80% training and 20% test and use GridSearchCV on my training data (80%) with GridSearchCV to find parameters and then evaluate my model with test data?

$\endgroup$
1
$\begingroup$

To test my model should I split my data into 3: training, validation, test?

If you have sufficient data to do so, then this might be reasonable. I think there may be some arguments to be made that cross validation is preferable to a one time data splitting. See Frank Harrell's Regression Modelling Strategies chapter. 5.3.4 for more if you are interested in those. In any case, the training data would be used to fit the model, and the validation data would be used to check the trained model's performance on new data. Once you select the best model (i.e. lowest loss on validation set), your final estimate of performance comes from the test set.

Does this mean that by using GridSearchCV I only need to split data into training and test?

Correct. Split the data into training and test, and then cross validation will split the data into folds, in which each fold acts as a validation set one time.

Should I divide my data into 80% training and 20% test and use GridSearchCV on my training data (80%) with GridSearchCV to find parameters and then evaluate my model with test data?

Plenty of people do that, yes.

My preference -- and there is no need for you to do this -- is to calculate the size of test by asking to what kind of precision I would like to know the out of sample performance. Using the central limit theorem, if I want a confidence interval which is $2d$ units long, then the number of samples I need is

$$ n = \Big( \dfrac{2 \sigma}{d} \Big)^2$$

where $\sigma$ is an estimate of the noise of the data. 80/20 is a bit arbitrary, and this way atleast gives you a defensible rationale for the split you make. Then, I would do nested cross validation as seen here. The benefit of nested cross validation is that is validates the procedure of selecting a model through grid search cross validation. In my experience, the estimates of out of sample performance are more faithful.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.