3
$\begingroup$

I have been trying to fit some data with a linear regression. I don't have any theoretical assumptions on the regression, and from what I know both an intercept or no intercept can be plausible results.

In the evaluation of the goodness of fit, I came across two different definitions of $R^2$, the most general one

$$ R^2 = 1 - \frac{SS_{res}}{SS_{tot}} $$

where $SS_{res}$ is the sum of squares of residuals and $SS_{tot}$ is the total sum of squares, and another definition that is justified only in the case of a fit with intercept

$$ R^2 = \frac{SS_{reg}}{SS_{tot}} $$

where $SS_{reg}$ is the regression sum of squares and the condition $SS_{tot} = SS_{reg} + SS_{res}$ is verified.

I know that some softwares (like R) use the second definition, with a "modified" version of $R^2$ in the case of no intercept, that can give conflictual results and this is a problem that has been already addressed. In another question I read that also for the first definition I should use a modified version of $R^2$ in the case of no intercept.

What I want to understand is if there is a general definition of $R^2$ that is always applicable? If there isn't one, in what way can I compare the goodness of a fit with intercept and a fit without intercept, and ultimately to choose between the two? Thank you in advance

$\endgroup$
1
  • 2
    $\begingroup$ In order to compare goodness of fit, it's enough to just compute $\mathrm{SS}_\mathrm{res}$, the residual sum of squares. This can easily be computed regardless of whether intercept is in the model. The difference in $R^2$ definitions only arises because of the different choices of normalization constant. $\endgroup$
    – amoeba
    Oct 30, 2015 at 11:03

1 Answer 1

1
$\begingroup$

$R^2$ compares your model to a naïve model that always predicts the same value. When your model includes an intercept, it makes sense that the naïve guess, ignoring the features, of the conditional mean would be the pooled/marginal mean of all $y$ observations: $\bar y$.

If we ignore the features, the linear regression equation is:

$$ y = \beta_0 + 0x_1 +\cdots + 0x_p =\beta_0 $$

OLS regression gives us $\hat\beta_0=\bar y$.

This is how we wind up with the denominator of the usual $R^2$ equation. Every prediction is the same $\bar y$ value.

$$ R^2_{usual} = 1-\dfrac{ \sum\bigg( y_i - \hat y_i \bigg)^2 }{ \sum\bigg( y_i - \bar y \bigg)^2 } $$

If we exclude an intercept from the model, then the model with no features is: $$ y = 0x_1 +\cdots + 0x_p = 0 $$

That is, the naïve model always predicts zero, and the comparison to the naïve model would reflect that fact.

$$ R^2_{no\_intercept} = 1-\dfrac{ \sum\bigg( y_i - \hat y_i \bigg)^2 }{ \sum\bigg( y_i - 0 \bigg)^2 } $$

If you want a general way of thinking about $R^2$, I would go with the idea of comparing to a naïve model that has no features.

It doesn't stop there. While this will wreck the typical interpretation of $R^2$ as the "proportion of variance explained" (which does not apply to the no-intercept version, anyway, and this interpretation vanishes in many settings with intercepts, too), you can use any baseline model in the denominator. Want to compare your fancy neural network to a linear model?

$$ R^2_{sorta} = 1-\dfrac{ \sum\bigg( y_i - \hat y_{i, neural} \bigg)^2 }{ \sum\bigg( y_i - \hat y_{i, linear} \bigg)^2 } $$

This is equivalent to comparing the models on mean squared error, $MSE = \frac{1}{n}\sum\big( y_i - \hat y_i \big)^2$, however, and I do not see an advantage to $R^2_{sorta}$ except that it forces you to make this comparison.

Viewing the problem in terms of $R^2_{sorta}$, the $R^2_{usual}$ and $R^2_{no\_intercept}$ equations are comparing to baseline models that are totally naïve.

Cardinal wrote a rather excellent answer related to this topic.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.