# What is this measure called: MSE divided by variance of dependent variable

When evaluating a regression model with cross-validation I thought that the meaningful measure would be MSE divided by the MSE of the null model which consists of always predicting the mean, $\frac{\hat E[(y-\hat{y})^2]}{\hat E[(y-\bar{y})^2]}$. This is 1 if the model does not add anything, and 0 if the prediction is perfect (and can even be greater than 1 if the model is actively harmful), To make it more interpretable, I can flip it around:

$1 - \frac{\hat E[(y-\hat{y})^2]}{\hat E[(y-\bar{y})^2]}$.

This can even be negative if the prediction is worse than the null model.

I have seen people use

$1 - \frac{\rm{var}(y-\hat{y})}{\rm{var}(y)}$

and calling it explained variance, but this seems too generous for the model as it does not penalize it for additive or multiplicative biases.

What is the measure I have above called? Is there a reason why it is not used or have I just missed the relevant examples?

• Since a regression model with an intercept term can't have additive bias, the additive bias issue is usually not a problem. The 'explained variance' one is usually called $R^2$, and in regression models that have an intercept, your measure will give the same outcome as that one. Commented Feb 22, 2014 at 10:17
• Note that the measure you have defined is not restricted to the range $(0, 1)$ - it is quite easily possible for the null model to have lower prediction error. Commented Feb 22, 2014 at 10:51
• Ah, this may be the issue: with cross-validation evaluation, your model can have a bias. Additionally, with leave-one-out cross-validation, the null model has perfect -1 correlation with the input (R² = 1). Reporting R² is then pretty meaningless. Commented Feb 22, 2014 at 11:32
• @probabilityislogic yes (and this does happen in practice). I made my question clearer now. Commented Feb 22, 2014 at 11:33
• What are you trying to accomplish by scoring your models this way? If you are trying to compare different models, then this measure will not rank them any differently from MSE alone because you are dividing all of the models' respective MSEs by the same quantity. If you combine the MSEs of your different models into a vector, transformation to your suggested metric is equivalent to multiplying the vector by a scalar. Maybe I'm misinterpreting what you're doing here, but it looks to me like the denominator of your metric is the same for all models. Commented Mar 4, 2014 at 15:54

Thanks to Mario Figueiredo who commented on my blog, I found out the answer: This measure is called $R^2_{CV}$ or $Q^2$.