I am using sklearn to train two models and compare their outcome with each other but I am not sure how to evaluate the models. As I have little data (approx. 300 data points) I want to use cross validation to train my model. So I have read that it is only used for hyper parameter tuning which I am already doing with GridSearchCV.

To train my model with more data I would want to use cross validation as well for training purposes. Does it make sense to do so?

My division of data would look like this:

  1. 80% of data: train/validation with cross validation and hyperparameter tuning (so the model is trained as well already in this step)
  2. 20% of data: test set and measuring MAE and RMSE as well as analysing prediction made by trained model

or should I do it that way before using GridSearch:

Split data into three data chunks:

  1. 60% of data for training
  2. 20% of data for testing
  3. 20% of data for validation

First step: Use the validation set in GridSearch and find the best parameters based on validation.

Second step: Train model with best parameters found by grid search with training set

Third step: Test model and calculate MAE and RSME on test set and as well as analysing prediction made by trained model

If both evaluation methods are not right, I would be grateful for any tips how to find the best way to evaluate my models.


1 Answer 1

  1. Divide your data into training set and test set (e.g. 80%, 20%).
  2. Perform the Grid Search (which uses k-fold cross validation) on the training set for two models and pick best parameters. GridSearchCV will divide the training set into $k$ train/validate splits on its own.
  3. Use the test set to compare the final models.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.