K-fold validation, how to use MSE and STD for model selection

Question

When using K-fold validation for model selection I'm wondering what's the best approach to select a model using both the mean square error (MSE) and the standard deviation of errors among folds (STD).

Below I present the approach I follow. I would like to know if is a good approach, if the assumptions are correct and what other better approaches could be followed.

Basic Ideas

In general I assume these ideas:

MSE average between folds will give a good estimate of the generalization error of the model. A low MSE average between folds will indicate that the model bias is low. The standard deviation of the MSE among folds can be use as a good approach to understand variance.

Proposed Approach

If a model gives the lowest MSE and STD then pick it (is the model with lowest bias and variance) If no model has the lowest values for MSE and STD, then pick the model that has the lowest bias (and the lowest variance) count, when compared to other models. This will be obtained in the following way: having models A,B,C and average MSE and STD values AMSE(A),ASTD(A), etc.., then A is the best model if

sum(AMSE(A)<AMSE(B),AMSE(A)<AMSE(C))+sum(ASTD(A)<ASTD(B),ASTD(A)<ASTD(C))

where true counts for 1 and false counts for 0. If from domain knowledge it is known that the data will tend models to high variance or to high bias, then pick the model with lowest STD, or lowest MSE respectively.

What are the best approaches to follow? are these assumptions ok?

Answer 1

Nope; you can reason about bias and variance by comparing predictions with the true value, CV folds have nothing to do with that.

What you can do with MSE variance over CV folds is to use it to test weather the difference between averages is significant, thus whether it is even justified to say that they are not equivalent.