I'm currently building an XGBoost model to predict sales for a certain line of products. I'm using Caret's train function with 10-fold cross validation to fine tune the model's hyper-parameters. The issue I currently face is that I only have 24 data points to work with, so I'm experience variance issues.
At the moment, I'm trying to determine which features I should add to the model and I don't know how to go about it. That's because each possible model's performance on the test set is variable when changing seeds. For example, with a seed of 777, one specific model has a test RMSE of 140, but on a different seed it has a test RMSE of 400.
Exactly how should I go about selecting the best model? My idea was to use the model with the lowest Training RMSE derived from the 10-fold cross validation. Any ideas?