3
$\begingroup$

For simple linear regression (SLR), in order for $R^2$ (the coefficient of determination) to be a meaningful measure, it must be true that $X$ and $Y$ are linearly correlated. Specifically, $R^2=r^2$, where $r$ is Pearson's correlation coefficient.

When we move into the multiple linear regression (MLR) framework, I'm curious how this linearity requirement transfers.

Take, for example, a polynomial model where $\hat y = \hat \beta_0 + \hat \beta_1x+ \hat \beta_2x^2$. In this model, assume the regression assumptions are met (i.e., the data are truly related according to a parabolic curve, so fitting $x^2$ as a predictor allows us to meet the linearity assumption).

Now, $X$ and $Y$ are not linearly related, but $X$ and $X^2$ jointly allow us to correctly model the association with $Y$. So, since we've met the linearity assumption of MLR, $R^2$ is meaningful, correct?

So would the conclusion be that $R^2$ is meaningful if the modeled relationship between $Y$ and the predictors (whatever they may be (i.e., even if they're polynomials)) satisfies the regression assumptions?

If so, we would say: In the case of SLR, this forces the requirement of $X$ and $Y$ being linearly related, but for MLR, the relationship between $\textbf X$ and $Y$ may be curved, as long as the linearity assumption is met.

$\endgroup$
  • $\begingroup$ I can't create new tags due to my reputation being < 300, so if anyone would like to create R2, linearity and/or linearity-assumption, they seem reasonable. $\endgroup$ – Meg Nov 1 '14 at 15:21
  • $\begingroup$ I added the tag for [r-squared]. There is a tag for [linear-model], but note that this means linear in the parameters (ie, the parameters of the model are coefficients (see here); "linearity" in the sense of rectilinear isn't really an assumption of regression. $\endgroup$ – gung Nov 1 '14 at 15:39
  • $\begingroup$ Thanks for adding r-squared. I was hoping to add a tag about linearity, indicating this problem deals with the linearity assumption (i.e., linear in the parameters) of regression. It also happens to deal with the "other" definition of linear, in that we need $X$ and $Y$ to be linear in order to meet the linearity assumption in SLR. I think adding the linear-model tag seems appropriate. $\endgroup$ – Meg Nov 1 '14 at 15:57
1
$\begingroup$

Coefficient of determination in multiple linear regression: Firstly, it is not correct to say $R^2$ is "meaningless" if it is zero. In a simple linear regression this just means that the response and explanatory vectors are uncorrelated, so it is very meaningful indeed. More generally, in multiple linear regression the coefficient-of-determination can be written in terms of the correlations for the variables using the quadratic form:

$$R^2 = \boldsymbol{r}_{\mathbf{y},\mathbf{x}}^\text{T} \boldsymbol{r}_{\mathbf{x},\mathbf{x}}^{-1} \boldsymbol{r}_{\mathbf{y},\mathbf{x}},$$

where $\boldsymbol{r}_{\mathbf{y},\mathbf{x}}$ is the vector of correlations between the response vector and each of the explanatory vectors, and $\boldsymbol{r}_{\mathbf{x},\mathbf{x}}$ is the matrix of correlations between the explanatory vectors (for more on this, see this related question). If the regression model form is correct then this statistic is meaningful, insofar as it gives a measure of goodness-of-fit which derives from the underlying correlations between the variables. If it is equal to zero then that gives a particular meaning, but it does not render the statistic meaningless.


Quadratic regression with no linear relationship: In the special case of the quadratic model you have specified, if there is no linear relationship, but there is a quadratic relationship then you have:

$$\boldsymbol{r}_{\mathbf{y},\mathbf{x}} = \begin{bmatrix} 0 \\ r_2 \end{bmatrix} \quad \quad \quad \boldsymbol{r}_{\mathbf{x},\mathbf{x}} = \begin{bmatrix} 1 & r_{1,2} \\ r_{1,2} & 1 \end{bmatrix}.$$

This gives you:

$$R^2 = \boldsymbol{r}_{\mathbf{y},\mathbf{x}}^\text{T} \boldsymbol{r}_{\mathbf{x},\mathbf{x}}^{-1} \boldsymbol{r}_{\mathbf{y},\mathbf{x}} = r_2^2 = \frac{(\sum (y_i - \bar{y}) (x_i^2 - \overline{x^2}))^2}{(\sum (y_i - \bar{y})^2) (\sum (x_i^2 - \overline{x^2})^2)}.$$

In this case the coefficient-of-determination is equal to the square of the correlation between the response vector $\mathbf{y}$ and the quadratic explanatory vector $\mathbf{x}^2$.

$\endgroup$
  • $\begingroup$ I'm not sure what the first half of your answer is regarding, as my question doesn't purport that $R^2$ is meaningless if it is 0. This is a question about when the assumptions of $R^2$ are met, so that its value - whatever it may be (0 or otherwise) - may be trusted ("is meaningful"). As with any statistic, its estimated value is meaningless if its theoretical assumptions are not sufficiently met in practice. $\endgroup$ – Meg Aug 31 '18 at 14:06
  • $\begingroup$ Perhaps I've misunderstood what you intended, but your first sentence claims that for $R^2$ to be a meaningful measure there must be correlation between $X$ and $Y$. That is what I was responding to. The issue here is that there are no assumptions required for $R^2$ to be a well-defined and meaningful measure. If the regression assumptions are met then this quantity will have a particular stochastic behaviour, but regardless of this, it is a measure of the squared multiple correlation in the problem. $\endgroup$ – Ben Sep 1 '18 at 2:01
  • $\begingroup$ If you can be more specific about what behavioural properties you would like $R^2$ to have for this statistic to be "trusted" then perhaps we can assist in saying what assumptions are required for that behaviour. $\endgroup$ – Ben Sep 1 '18 at 2:02
  • $\begingroup$ Thanks for clarifying. My first sentence says, "For simple linear regression (SLR), in order for $R^2$...to be a meaningful measure, it must be true that $X$ and $Y$ are linearly correlated." The keyword is linearly - if they are not linearly related, this measure is not meaningful, because a value of 0 doesn't tell us there's no correlation, just no linear correlation ($X$ and $Y$ could be strongly related via a parabolic relationship, e.g., which this measure would miss). This question is about how that assumption translates into higher-level models. $\endgroup$ – Meg Sep 2 '18 at 15:58
  • $\begingroup$ Very simplistically: If the linearity assumption of a regression model (whether it be simple or multiple (including polynomial)) is met, does $R^2$ always have the same meaning? $\endgroup$ – Meg Sep 2 '18 at 16:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.