When using K-fold validation for model selection I'm wondering what's the best approach to select a model using both the mean square error (MSE) and the standard deviation of errors among folds (STD).
Below I present the approach I follow. I would like to know if is a good approach, if the assumptions are correct and what other better approaches could be followed.
Basic Ideas
In general I assume these ideas:
MSE average between folds will give a good estimate of the generalization error of the model. A low MSE average between folds will indicate that the model bias is low. The standard deviation of the MSE among folds can be use as a good approach to understand variance.
Proposed Approach
If a model gives the lowest MSE and STD then pick it (is the model with lowest bias and variance) If no model has the lowest values for MSE and STD, then pick the model that has the lowest bias (and the lowest variance) count, when compared to other models. This will be obtained in the following way: having models A,B,C and average MSE and STD values AMSE(A),ASTD(A), etc.., then A is the best model if
sum(AMSE(A)<AMSE(B),AMSE(A)<AMSE(C))+sum(ASTD(A)<ASTD(B),ASTD(A)<ASTD(C))
where true counts for 1 and false counts for 0. If from domain knowledge it is known that the data will tend models to high variance or to high bias, then pick the model with lowest STD, or lowest MSE respectively.
What are the best approaches to follow? are these assumptions ok?