$R^2$ (coefficient of determination) and linearity in multiple linear regression

Question

For simple linear regression (SLR), in order for $R^2$ (the coefficient of determination) to be a meaningful measure, it must be true that $X$ and $Y$ are linearly correlated. Specifically, $R^2=r^2$, where $r$ is Pearson's correlation coefficient.

When we move into the multiple linear regression (MLR) framework, I'm curious how this linearity requirement transfers.

Take, for example, a polynomial model where $\hat y = \hat \beta_0 + \hat \beta_1x+ \hat \beta_2x^2$. In this model, assume the regression assumptions are met (i.e., the data are truly related according to a parabolic curve, so fitting $x^2$ as a predictor allows us to meet the linearity assumption).

Now, $X$ and $Y$ are not linearly related, but $X$ and $X^2$ jointly allow us to correctly model the association with $Y$. So, since we've met the linearity assumption of MLR, $R^2$ is meaningful, correct?

So would the conclusion be that $R^2$ is meaningful if the modeled relationship between $Y$ and the predictors (whatever they may be (i.e., even if they're polynomials)) satisfies the regression assumptions?

If so, we would say: In the case of SLR, this forces the requirement of $X$ and $Y$ being linearly related, but for MLR, the relationship between $\textbf X$ and $Y$ may be curved, as long as the linearity assumption is met.

Answer 1

Coefficient of determination in multiple linear regression: Firstly, it is not correct to say $R^2$ is "meaningless" if it is zero. In a simple linear regression this just means that the response and explanatory vectors are uncorrelated, so it is very meaningful indeed. More generally, in multiple linear regression the coefficient-of-determination can be written in terms of the correlations for the variables using the quadratic form:

$$R^2 = \boldsymbol{r}_{\mathbf{y},\mathbf{x}}^\text{T} \boldsymbol{r}_{\mathbf{x},\mathbf{x}}^{-1} \boldsymbol{r}_{\mathbf{y},\mathbf{x}},$$

where $\boldsymbol{r}_{\mathbf{y},\mathbf{x}}$ is the vector of correlations between the response vector and each of the explanatory vectors, and $\boldsymbol{r}_{\mathbf{x},\mathbf{x}}$ is the matrix of correlations between the explanatory vectors (for more on this, see this related question). If the regression model form is correct then this statistic is meaningful, insofar as it gives a measure of goodness-of-fit which derives from the underlying correlations between the variables. If it is equal to zero then that gives a particular meaning, but it does not render the statistic meaningless.

---

Quadratic regression with no linear relationship: In the special case of the quadratic model you have specified, if there is no linear relationship, but there is a quadratic relationship then you have:

$$\boldsymbol{r}_{\mathbf{y},\mathbf{x}} = \begin{bmatrix} 0 \\ r_2 \end{bmatrix} \quad \quad \quad \boldsymbol{r}_{\mathbf{x},\mathbf{x}} = \begin{bmatrix} 1 & r_{1,2} \\ r_{1,2} & 1 \end{bmatrix}.$$

This gives you:

$$R^2 = \boldsymbol{r}_{\mathbf{y},\mathbf{x}}^\text{T} \boldsymbol{r}_{\mathbf{x},\mathbf{x}}^{-1} \boldsymbol{r}_{\mathbf{y},\mathbf{x}} = r_2^2 = \frac{(\sum (y_i - \bar{y}) (x_i^2 - \overline{x^2}))^2}{(\sum (y_i - \bar{y})^2) (\sum (x_i^2 - \overline{x^2})^2)}.$$

In this case the coefficient-of-determination is equal to the square of the correlation between the response vector $\mathbf{y}$ and the quadratic explanatory vector $\mathbf{x}^2$.

Answer 2

The nonlinear terms are new variables.

You write your model as $\hat y = \hat \beta_0 + \hat \beta_1x+ \hat \beta_2x^2$. Instead, define $x_1 := x$ and $x_2 := x^2$. Now $\hat y = \hat \beta_0 + \hat \beta_1x_1+ \hat \beta_2x_2$, and that looks linear, doesn't it?

The math does not have to know how you got $x_2$. All the model cares about are the values of your features. One you measured; one you calculated. The model just sees numbers.