I'm running a Grid Search to find the optimal parameters for xgboost via sklearn.
I can see that the grid search picks the set of parameters with lowest mean MSE.
The problem is that upon inspecting the standard deviations they are in the range of the mean, which suggests that there is not statistical significance between the choice of parameters.
Can a means difference test be ran on these means? How do we make sure there is a statistically significant difference between the means?
And I find it weird that these packages work picking the lowest mean without paying attention to the standard deviation. Do you know any package that looks into this issue?