Choosing the Best Performing Model when Test Set MSE's Are Highly Variable? I'm currently building an XGBoost model to predict sales for a certain line of products. I'm using Caret's train function with 10-fold cross validation to fine tune the model's hyper-parameters. The issue I currently face is that I only have 24 data points to work with, so I'm experience variance issues.
At the moment, I'm trying to determine which features I should add to the model and I don't know how to go about it. That's because each possible model's performance on the test set is variable when changing seeds. For example, with a seed of 777, one specific model has a test RMSE of 140, but on a different seed it has a test RMSE of 400.
Exactly how should I go about selecting the best model? My idea was to use the model with the lowest Training RMSE derived from the 10-fold cross validation. Any ideas?
 A: The "one standard error rule" is commonly recommended. I'll just copy over what I wrote at that thread. Click over there for references and other views.

Assume we consider models $M_\tau$ indexed by a complexity parameter $\tau\in\mathbb{R}$, such that $M_\tau$ is "more complex" than $M_{\tau'}$ exactly when $\tau>\tau'$. Assume further that we assess the quality of a model $M$ by some randomization process, e.g., cross-validation. Let $q(M)$ denote the "average" quality of $M$, e.g., the mean out-of-bag prediction error across many cross-validation runs. We wish to minimize this quantity.
However, since our quality measure comes from some randomization procedure, it comes with variability. Let $s(M)$ denote the standard error of the quality of $M$ across the randomization runs, e.g., the standard deviation of the out-of-bag prediction error of $M$ over cross-validation runs.
Then we choose the model $M_\tau$, where $\tau$ is the smallest $\tau$ such that
$$q(M_\tau)\leq q(M_{\tau'})+s(M_{\tau'}),$$
where $\tau'$ indexes the (on average) best model, $q(M_{\tau'})=\min_\tau q(M_\tau)$.
That is, we choose the simplest model (the smallest $\tau$) which is no more than one standard error worse than the best model $M_{\tau'}$ in the randomization procedure.
A: I would fit a Gaussian Process or similar to your validation errors and then select hyper-parameters at either the maximum of the fitted GP or where the fitted mean + 1 standard deviation has a maximum.
You might be able to reduce you variance by running a repeated K-fold cross validation instead of just a single repetition. Repeated K-fold consists of normal K-fold cross validation where you then shuffle the data and do another K-fold etc. So for 3 repeats of 5 fold CV, you would have a total of 15 folds.
