I'm running xgboost on some simulation, where my sample size is 125. I was measuring the 5-fold cross validation error, i.e., in each round my training sample size is 100 and testing sample size is 25. Assuming all other parameters are fixed but the "n_estimators", i.e., the number of boosting rounds.
I have two options:
run the 5-fold cv for different n_estimators and do not use early stopping. In each fold, use all 100 obs as training and no validation for early stopping--in this case, I may choose the best n_estimator from the cv results;
run the 5-fold cv, in each fold, further split the training sample into training (80) and validation (20), train the model on the 80 training observations and monitor early stopping on the 20 validation observations--in this case I may select a huge n_estimator and let it auto stop.
The questions are
In option 1, if I have another separate testing sample, can I use the 5 cross-validation models on the testing data and compute the average/ majority vote? Or do I need to train the model again with the best parameters on all 125 obs and make prediction on the testing set?
In option 2, is the 80 training obs enough to train the model/ 20 validation obs enough to monitor the performance? (in option 1 we also have a small sample size, but a little better)
Which option is better at comparing the xgboost model with other models?