# Using the standard deviation in Cross Validation

I'm running a Grid Search to find the optimal parameters for xgboost via sklearn.

I can see that the grid search picks the set of parameters with lowest mean MSE.

The problem is that upon inspecting the standard deviations they are in the range of the mean, which suggests that there is not statistical significance between the choice of parameters.

Can a means difference test be ran on these means? How do we make sure there is a statistically significant difference between the means?

And I find it weird that these packages work picking the lowest mean without paying attention to the standard deviation. Do you know any package that looks into this issue?

• I suggest you look at the 1 standard deviation rule used in CV. It basically tells you to chose a simple model as long as you are within 1 standard deviation of the minimum. It is based on the problem you are mentioning. stat.cmu.edu/~ryantibs/datamining/lectures/18-val1.pdf Oct 19 '18 at 8:25
• Thanks @CowboyTrader, this is in fact related to the problem I mentioned, although in my particular set up, the parameter space has more variables, and I was trying only some values for those variables, and all the values would be within their standard deviation which makes de search difficult. Maybe I can try one parameter at a time and trying to visualize the curve like on the slides you suggest. In the worse case where there is no value outside one the standard deviation of the others, should I give up on the optimization? Oct 19 '18 at 12:38

"How do we make sure there is a statistically significant difference between the means?...these packages work picking the lowest mean without paying attention to the standard deviation..."

Searches for parameter values are looking for an optimum solution, based on specified criteria, in a particular learning/modeling task. There is no assurance that the parameter values at the optimum solution can be distinguished statistically from other sets of parameter values in the search space. One could argue that with finite data sets and smooth optimization criteria it would be impossible to find "significant" differences between the identified optimum values and sets of parameter values that are sufficiently close to that optimum.

What matters is how well the model based on the chosen parameter values performs for the purpose that you have in mind. You will be better served by focusing more on the significance of the model performance overall (via cross-validation, bootstrapping, etc.) and paying less attention to whether your optimum set of parameter values has a "statistically significant" difference from some other set of values.

Let me add some points to @EdM's excellent answer (which are too long for a comment).

And I find it weird that these packages work picking the lowest mean without paying attention to the standard deviation.

• I totally agree with you (OP) knowing uncertainty could (IMHO should) be used to stop the optimization process: no point in trying to tie down the optimum within a zone where we cannot distinguish performance/target function values.

• From a modeling point of view, of course, EdM's "sufficiently close to the optimum" should hopefully be a comfortably large zone so even if we have uncertainty about the true optimum we can assume that our opimized model isn't doing too much worse than the model fit using the unknown true optimum of hyperparameters.

• Of course, we can also turn around the question a bit. We do have (at least) two sources of variance to our performance evaluation: model instability and the finite number of cases we test. Model instability is IMHO something that crucially needs to be addressed during optimization (can be done e.g. by the 1-sd-rule mentioned above). Finite number of test cases is something we can influence rather directly, and IMHO we should check at the very least that the sample size we have allows to distinguish statistically what we consider practically significant differences in performance wrt. the application at hand.

Do you know any package that looks into this issue?

No, but I've been working on and off (as time permits/optimizations are needed) towards this. In R, though. I'd be happy to join forces if you like.

• One variable at a time is not going to work unless you know the hyperparameter in question does not interact with other of your hyperparameters. Which is rare, but occasionally the case (random forest number of trees comes to my mind).
• Thanks @cbeleites! Great to know you are working on this. I'd like to collaborate but not that strong in R and getting started in Machine Learning, not sure if I can be of much help. Oct 24 '18 at 16:38