5
$\begingroup$

When evaluating a regression model with cross-validation I thought that the meaningful measure would be MSE divided by the MSE of the null model which consists of always predicting the mean, $\frac{\hat E[(y-\hat{y})^2]}{\hat E[(y-\bar{y})^2]}$. This is 1 if the model does not add anything, and 0 if the prediction is perfect (and can even be greater than 1 if the model is actively harmful), To make it more interpretable, I can flip it around:

$1 - \frac{\hat E[(y-\hat{y})^2]}{\hat E[(y-\bar{y})^2]}$.

This can even be negative if the prediction is worse than the null model.

I have seen people use

$1 - \frac{\rm{var}(y-\hat{y})}{\rm{var}(y)}$

and calling it explained variance, but this seems too generous for the model as it does not penalize it for additive or multiplicative biases.

What is the measure I have above called? Is there a reason why it is not used or have I just missed the relevant examples?

Improve this question
$\endgroup$
10
  • $\begingroup$ Since a regression model with an intercept term can't have additive bias, the additive bias issue is usually not a problem. The 'explained variance' one is usually called $R^2$, and in regression models that have an intercept, your measure will give the same outcome as that one. $\endgroup$
    – Glen_b
    Commented Feb 22, 2014 at 10:17
  • $\begingroup$ Note that the measure you have defined is not restricted to the range $(0, 1) $ - it is quite easily possible for the null model to have lower prediction error. $\endgroup$
    – probabilityislogic
    Commented Feb 22, 2014 at 10:51
  • $\begingroup$ Ah, this may be the issue: with cross-validation evaluation, your model can have a bias. Additionally, with leave-one-out cross-validation, the null model has perfect -1 correlation with the input (R² = 1). Reporting R² is then pretty meaningless. $\endgroup$
    – luispedro
    Commented Feb 22, 2014 at 11:32
  • $\begingroup$ @probabilityislogic yes (and this does happen in practice). I made my question clearer now. $\endgroup$
    – luispedro
    Commented Feb 22, 2014 at 11:33
  • $\begingroup$ What are you trying to accomplish by scoring your models this way? If you are trying to compare different models, then this measure will not rank them any differently from MSE alone because you are dividing all of the models' respective MSEs by the same quantity. If you combine the MSEs of your different models into a vector, transformation to your suggested metric is equivalent to multiplying the vector by a scalar. Maybe I'm misinterpreting what you're doing here, but it looks to me like the denominator of your metric is the same for all models. $\endgroup$
    – David Marx
    Commented Mar 4, 2014 at 15:54

1 Answer 1

Reset to default
4
$\begingroup$

Thanks to Mario Figueiredo who commented on my blog, I found out the answer: This measure is called $R^2_{CV}$ or $Q^2$.

Reference: The Prediction Sum of Squares as a General Measure for Regression Diagnostics Nguyen T. Quan Journal of Business & Economic Statistics Vol. 6, No. 4 (Oct., 1988), pp. 501-504 JSTOR

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.