5
$\begingroup$

For simple linear regression (SLR), in order for $R^2$ (the coefficient of determination) to be a meaningful measure, it must be true that $X$ and $Y$ are linearly correlated. Specifically, $R^2=r^2$, where $r$ is Pearson's correlation coefficient.

When we move into the multiple linear regression (MLR) framework, I'm curious how this linearity requirement transfers.

Take, for example, a polynomial model where $\hat y = \hat \beta_0 + \hat \beta_1x+ \hat \beta_2x^2$. In this model, assume the regression assumptions are met (i.e., the data are truly related according to a parabolic curve, so fitting $x^2$ as a predictor allows us to meet the linearity assumption).

Now, $X$ and $Y$ are not linearly related, but $X$ and $X^2$ jointly allow us to correctly model the association with $Y$. So, since we've met the linearity assumption of MLR, $R^2$ is meaningful, correct?

So would the conclusion be that $R^2$ is meaningful if the modeled relationship between $Y$ and the predictors (whatever they may be (i.e., even if they're polynomials)) satisfies the regression assumptions?

If so, we would say: In the case of SLR, this forces the requirement of $X$ and $Y$ being linearly related, but for MLR, the relationship between $\textbf X$ and $Y$ may be curved, as long as the linearity assumption is met.

$\endgroup$
4
  • $\begingroup$ I can't create new tags due to my reputation being < 300, so if anyone would like to create R2, linearity and/or linearity-assumption, they seem reasonable. $\endgroup$
    – Meg
    Nov 1, 2014 at 15:21
  • $\begingroup$ I added the tag for [r-squared]. There is a tag for [linear-model], but note that this means linear in the parameters (ie, the parameters of the model are coefficients (see here); "linearity" in the sense of rectilinear isn't really an assumption of regression. $\endgroup$ Nov 1, 2014 at 15:39
  • $\begingroup$ Thanks for adding r-squared. I was hoping to add a tag about linearity, indicating this problem deals with the linearity assumption (i.e., linear in the parameters) of regression. It also happens to deal with the "other" definition of linear, in that we need $X$ and $Y$ to be linear in order to meet the linearity assumption in SLR. I think adding the linear-model tag seems appropriate. $\endgroup$
    – Meg
    Nov 1, 2014 at 15:57
  • $\begingroup$ One resolution of this issue is to note that $R^2$ is the square of the correlation coefficient between the fitted and actual values. This reduces the question to whether the predicted and fitted values appear to follow a linear relationship -- regardless of how nonlinear the relationship among the explanatory variables might be. Some people might find the illustrations I posted at stats.stackexchange.com/a/354256/919 to be helpful, too. $\endgroup$
    – whuber
    Jun 23, 2021 at 16:55

2 Answers 2

1
$\begingroup$

Coefficient of determination in multiple linear regression: Firstly, it is not correct to say $R^2$ is "meaningless" if it is zero. In a simple linear regression this just means that the response and explanatory vectors are uncorrelated, so it is very meaningful indeed. More generally, in multiple linear regression the coefficient-of-determination can be written in terms of the correlations for the variables using the quadratic form:

$$R^2 = \boldsymbol{r}_{\mathbf{y},\mathbf{x}}^\text{T} \boldsymbol{r}_{\mathbf{x},\mathbf{x}}^{-1} \boldsymbol{r}_{\mathbf{y},\mathbf{x}},$$

where $\boldsymbol{r}_{\mathbf{y},\mathbf{x}}$ is the vector of correlations between the response vector and each of the explanatory vectors, and $\boldsymbol{r}_{\mathbf{x},\mathbf{x}}$ is the matrix of correlations between the explanatory vectors (for more on this, see this related question). If the regression model form is correct then this statistic is meaningful, insofar as it gives a measure of goodness-of-fit which derives from the underlying correlations between the variables. If it is equal to zero then that gives a particular meaning, but it does not render the statistic meaningless.


Quadratic regression with no linear relationship: In the special case of the quadratic model you have specified, if there is no linear relationship, but there is a quadratic relationship then you have:

$$\boldsymbol{r}_{\mathbf{y},\mathbf{x}} = \begin{bmatrix} 0 \\ r_2 \end{bmatrix} \quad \quad \quad \boldsymbol{r}_{\mathbf{x},\mathbf{x}} = \begin{bmatrix} 1 & r_{1,2} \\ r_{1,2} & 1 \end{bmatrix}.$$

This gives you:

$$R^2 = \boldsymbol{r}_{\mathbf{y},\mathbf{x}}^\text{T} \boldsymbol{r}_{\mathbf{x},\mathbf{x}}^{-1} \boldsymbol{r}_{\mathbf{y},\mathbf{x}} = r_2^2 = \frac{(\sum (y_i - \bar{y}) (x_i^2 - \overline{x^2}))^2}{(\sum (y_i - \bar{y})^2) (\sum (x_i^2 - \overline{x^2})^2)}.$$

In this case the coefficient-of-determination is equal to the square of the correlation between the response vector $\mathbf{y}$ and the quadratic explanatory vector $\mathbf{x}^2$.

$\endgroup$
7
  • $\begingroup$ I'm not sure what the first half of your answer is regarding, as my question doesn't purport that $R^2$ is meaningless if it is 0. This is a question about when the assumptions of $R^2$ are met, so that its value - whatever it may be (0 or otherwise) - may be trusted ("is meaningful"). As with any statistic, its estimated value is meaningless if its theoretical assumptions are not sufficiently met in practice. $\endgroup$
    – Meg
    Aug 31, 2018 at 14:06
  • $\begingroup$ Perhaps I've misunderstood what you intended, but your first sentence claims that for $R^2$ to be a meaningful measure there must be correlation between $X$ and $Y$. That is what I was responding to. The issue here is that there are no assumptions required for $R^2$ to be a well-defined and meaningful measure. If the regression assumptions are met then this quantity will have a particular stochastic behaviour, but regardless of this, it is a measure of the squared multiple correlation in the problem. $\endgroup$
    – Ben
    Sep 1, 2018 at 2:01
  • $\begingroup$ If you can be more specific about what behavioural properties you would like $R^2$ to have for this statistic to be "trusted" then perhaps we can assist in saying what assumptions are required for that behaviour. $\endgroup$
    – Ben
    Sep 1, 2018 at 2:02
  • $\begingroup$ Thanks for clarifying. My first sentence says, "For simple linear regression (SLR), in order for $R^2$...to be a meaningful measure, it must be true that $X$ and $Y$ are linearly correlated." The keyword is linearly - if they are not linearly related, this measure is not meaningful, because a value of 0 doesn't tell us there's no correlation, just no linear correlation ($X$ and $Y$ could be strongly related via a parabolic relationship, e.g., which this measure would miss). This question is about how that assumption translates into higher-level models. $\endgroup$
    – Meg
    Sep 2, 2018 at 15:58
  • $\begingroup$ Very simplistically: If the linearity assumption of a regression model (whether it be simple or multiple (including polynomial)) is met, does $R^2$ always have the same meaning? $\endgroup$
    – Meg
    Sep 2, 2018 at 16:05
0
$\begingroup$

The nonlinear terms are new variables.

You write your model as $\hat y = \hat \beta_0 + \hat \beta_1x+ \hat \beta_2x^2$. Instead, define $x_1 := x$ and $x_2 := x^2$. Now $\hat y = \hat \beta_0 + \hat \beta_1x_1+ \hat \beta_2x_2$, and that looks linear, doesn't it?

The math does not have to know how you got $x_2$. All the model cares about are the values of your features. One you measured; one you calculated. The model just sees numbers.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.