Normally, you have train, validation and test sets for training, tuning (hyperparameters) and finally evaluating a machine-learning model.
If we use cross-validation, then we can effectively have only train and test sets, where train set is used for CV (both training and tuning, using several folds). And after the best model is selected, it is trained on the whole train set and evaluated on test. Am I correct here?
Now, I am a bit confused about the boosting trees. Let's take LightGBM as an example.
When you are training a boosting tree, normally you set some value as an early_stopping_rounds and you do a separate CV to define the optimal number of trees in the model (e.g. with LGBM it is lightgbm.cv method).
Does it mean, that I need again three datasets?
- Train (to find the optimal number of trees using CV and then train on the whole train set, using the found optimal number of trees)
- Validation (tune other hyperparameters)
- Test (final evaluation)