4
$\begingroup$

When evaluating a regression model with cross-validation I thought that the meaningful measure would be MSE divided by the MSE of the null model which consists of always predicting the mean, $\frac{\hat E[(y-\hat{y})^2]}{\hat E[(y-\bar{y})^2]}$. This is 1 if the model does not add anything, and 0 if the prediction is perfect (and can even be greater than 1 if the model is actively harmful), To make it more interpretable, I can flip it around:

$1 - \frac{\hat E[(y-\hat{y})^2]}{\hat E[(y-\bar{y})^2]}$.

This can even be negative if the prediction is worse than the null model.

I have seen people use

$1 - \frac{\rm{var}(y-\hat{y})}{\rm{var}(y)}$

and calling it explained variance, but this seems too generous for the model as it does not penalize it for additive or multiplicative biases.

What is the measure I have above called? Is there a reason why it is not used or have I just missed the relevant examples?

$\endgroup$
  • $\begingroup$ Since a regression model with an intercept term can't have additive bias, the additive bias issue is usually not a problem. The 'explained variance' one is usually called $R^2$, and in regression models that have an intercept, your measure will give the same outcome as that one. $\endgroup$ – Glen_b -Reinstate Monica Feb 22 '14 at 10:17
  • $\begingroup$ Note that the measure you have defined is not restricted to the range $(0, 1) $ - it is quite easily possible for the null model to have lower prediction error. $\endgroup$ – probabilityislogic Feb 22 '14 at 10:51
  • $\begingroup$ Ah, this may be the issue: with cross-validation evaluation, your model can have a bias. Additionally, with leave-one-out cross-validation, the null model has perfect -1 correlation with the input (R² = 1). Reporting R² is then pretty meaningless. $\endgroup$ – luispedro Feb 22 '14 at 11:32
  • $\begingroup$ @probabilityislogic yes (and this does happen in practice). I made my question clearer now. $\endgroup$ – luispedro Feb 22 '14 at 11:33
  • $\begingroup$ What are you trying to accomplish by scoring your models this way? If you are trying to compare different models, then this measure will not rank them any differently from MSE alone because you are dividing all of the models' respective MSEs by the same quantity. If you combine the MSEs of your different models into a vector, transformation to your suggested metric is equivalent to multiplying the vector by a scalar. Maybe I'm misinterpreting what you're doing here, but it looks to me like the denominator of your metric is the same for all models. $\endgroup$ – David Marx Mar 4 '14 at 15:54
3
$\begingroup$

Thanks to Mario Figueiredo who commented on my blog, I found out the answer: This measure is called $R^2_{CV}$ or $Q^2$.

Reference: The Prediction Sum of Squares as a General Measure for Regression Diagnostics Nguyen T. Quan Journal of Business & Economic Statistics Vol. 6, No. 4 (Oct., 1988), pp. 501-504 JSTOR

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.