I am using
sklearn to train two models and compare their outcome with each other but I am not sure how to evaluate the models. As I have little data (approx. 300 data points) I want to use cross validation to train my model. So I have read that it is only used for hyper parameter tuning which I am already doing with
To train my model with more data I would want to use cross validation as well for training purposes. Does it make sense to do so?
My division of data would look like this:
- 80% of data: train/validation with cross validation and hyperparameter tuning (so the model is trained as well already in this step)
- 20% of data: test set and measuring MAE and RMSE as well as analysing prediction made by trained model
or should I do it that way before using GridSearch:
Split data into three data chunks:
- 60% of data for training
- 20% of data for testing
- 20% of data for validation
First step: Use the validation set in GridSearch and find the best parameters based on validation.
Second step: Train model with best parameters found by grid search with training set
Third step: Test model and calculate MAE and RSME on test set and as well as analysing prediction made by trained model
If both evaluation methods are not right, I would be grateful for any tips how to find the best way to evaluate my models.