1
$\begingroup$

When using K-fold validation for model selection I'm wondering what's the best approach to select a model using both the mean square error (MSE) and the standard deviation of errors among folds (STD).

Below I present the approach I follow. I would like to know if is a good approach, if the assumptions are correct and what other better approaches could be followed.

Basic Ideas

In general I assume these ideas:

MSE average between folds will give a good estimate of the generalization error of the model. A low MSE average between folds will indicate that the model bias is low. The standard deviation of the MSE among folds can be use as a good approach to understand variance.

Proposed Approach

If a model gives the lowest MSE and STD then pick it (is the model with lowest bias and variance) If no model has the lowest values for MSE and STD, then pick the model that has the lowest bias (and the lowest variance) count, when compared to other models. This will be obtained in the following way: having models A,B,C and average MSE and STD values AMSE(A),ASTD(A), etc.., then A is the best model if

sum(AMSE(A)<AMSE(B),AMSE(A)<AMSE(C))+sum(ASTD(A)<ASTD(B),ASTD(A)<ASTD(C))

where true counts for 1 and false counts for 0. If from domain knowledge it is known that the data will tend models to high variance or to high bias, then pick the model with lowest STD, or lowest MSE respectively.

What are the best approaches to follow? are these assumptions ok?

$\endgroup$
1
$\begingroup$

Nope; you can reason about bias and variance by comparing predictions with the true value, CV folds have nothing to do with that.

What you can do with MSE variance over CV folds is to use it to test weather the difference between averages is significant, thus whether it is even justified to say that they are not equivalent.

$\endgroup$
  • $\begingroup$ And what kind of analysis can be performed using the differences between averages of MSE and STD? Can I use this for model selection? $\endgroup$ – Javierfdr Jan 15 '15 at 1:12
  • $\begingroup$ You can do a Wilcoxon test or t-test (after appropriate normal distribution check) using per-fold MSE sets as compared samples; it is likely that it will tell you that models are statistically indistinguishable, thus that model selection in this context is pointless. $\endgroup$ – user88 Jan 15 '15 at 20:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.