# Is R-squared truly an invalid metric for non-linear models?

I have read that R-squared is invalid for non-linear models, because the relationship that SSR + SSE = SSTotal no longer holds. Can somebody explain why this is true?

SSR and SSE are just the squared norms of the regression and residual vectors, whose $i^{th}$ components are $(\hat{Y_i}-\bar{Y})$ and $(Y_i-\hat{Y_i})$, respectively. As long as these vectors are orthogonal to one another, then shouldn't the above relationship always hold, no matter what kind of function that is used to map predictor values to fitted ones?

Furthermore, shouldn't the regression and residual vectors associated with any least-squares model be orthogonal, by definition of least-squares? The residual vector is the difference between vector $(Y_i-\bar{Y_i})$ and the regression vector. If the regression vector is such that the residual/difference vector is not orthogonal to it, then the regression vector can be multiplied by a constant so that it is now orthogonal to the residual/difference vector. This should also reduce the norm of the residual/difference vector.

If I have explained this poorly, then please tell me and I will try to clarify.

• Since you can always compute $R^2,$ could you explain in what sense it could be considered "invalid"? For what purpose, exactly?
– whuber
Jul 31, 2018 at 11:43
• This hinges in part on how the quantity is defined. I see little harm in cautiously using square of correlation between observed and fitted response as a descriptive statistic, but it's not necessarily what nonlinear regression maximises. What is, or should be, key is that nonlinear regression makes use of some functional form with some scientific (engineering, medical, whatever) rationale or at least plausibility: that is a context which should define what measure of goodness or badness of fit is most useful. Jul 31, 2018 at 17:54
• @whuber Sorry I did not see your comment when it was originally posted. I think R-squared is considered invalid in nonlinear cases for multiple reasons, but I was mainly focusing on the claim that SSE + SSR =/= SSTotal when linearity is violated, because I believed it to be wrong.
– Greg
Aug 22, 2018 at 15:23
• Though it's a naive question I wonder about the term "linear" here. Can I use $R²$ for linear models like $"y=a+bx+cx²"$?
– Ben
Mar 9, 2022 at 14:49

The sums-of-squares in linear regression are special cases of the more general deviance values in the generalised linear model. In the more general model there is a response distribution with mean linked to a linear function of the explanatory variables (with an intercept term). The three deviance statistics in a GLM are defined as:

$$\begin{matrix} \text{Null Deviance} \quad \quad \text{ } \text{ } & & \text{ } D_{TOT} = 2(\hat{\ell}_{S} - \hat{\ell}_0), \\[6pt] \text{Explained Deviance} & & D_{REG} = 2(\hat{\ell}_{p} - \hat{\ell}_0), \\[6pt] \text{Residual Deviance}^\dagger \text{ } & & \text{ } D_{RES} = 2(\hat{\ell}_{S} - \hat{\ell}_{p}). \\[6pt] \end{matrix}$$

In these expressions the value $\hat{\ell}_S$ is the maximised log-likelihood under a saturated model (one parameter per data point), $\hat{\ell}_0$ is the maximised log-likelihood under a null model (intercept only), and $\hat{\ell}_{p}$ is the maximised log-likelihood under the model (intercept term and $p$ coefficients).

These deviance statistics play a role analogous to scaled versions of the sums-of-squares in linear regression. It is easy to see that they satisfy the decomposition $D_{TOT} = D_{REG} + D_{RES}$, which is analogous to the decomposition of the sums-of-squares in linear regression. In fact, in the case where you have a normal response distribution with a linear link function you get a linear regression model, and the deviance statistics reduce to the following:

\begin{aligned} D_{TOT} = \frac{1}{\sigma^2} \sum_{i=1}^n (y_i - \bar{y})^2 = \frac{1}{\sigma^2} \cdot SS_{TOT}, \\[6pt] D_{REG} = \frac{1}{\sigma^2} \sum_{i=1}^n (\hat{y}_i - \bar{y})^2 = \frac{1}{\sigma^2} \cdot SS_{REG}, \\[6pt] D_{RES} = \frac{1}{\sigma^2} \sum_{i=1}^n (y_i - \hat{y}_i)^2 = \frac{1}{\sigma^2} \cdot SS_{RES}. \\[6pt] \end{aligned}

Now, the coefficient of variation in a linear regression model is a goodness-of-fit statistic that measures the proportion of the total variation in the response that is attributable to the explanatory variables. A natural extension in the case of a GLM is to form the statistic:

$$R_{GLM}^2 = 1-\frac{D_{RES}}{D_{TOT}} = \frac{D_{REG}}{D_{TOT}}.$$

It is easily seen that this statistic reduces to the coefficient of variation in the special case of linear regression, since the scaling values cancel out. In the broader context of a GLM the statistic has a natural interpretation that is analogous to its interpretation in linear regression: it gives the proportion of the null deviance that is explained by the explanatory variables in the model.

Now that we have seen how the sums-of-squares in linear regression extend to the deviances in a GLM, we can see that the regular coefficient of variation is inappropriate in the non-linear model, since it is specific to the case of a linear model with a normally distributed error term. Nevertheless, we can see that although the standard coefficient of variation is inappropriate, it is possible to form an appropriate analogy using the deviance values, with an analogous interpretation.

$^\dagger$ The residual deviance is sometimes just called the deviance.

• Thanks for useful post. Does this generic R2 1-DRES/DTOT have a name btw? I sometimes see it quoted as McFadden's, but McFadden I believe was defined as 1-logLik(model)/logLik(null_model), which would only match the formula above is the logLik(saturated_model) was zero (which is the case for logistic regression, but not for other models). So does it have an accepted name? Jun 14, 2019 at 2:30
• Maybe one could call it "Deviance R2" and the one I suggest here, stats.stackexchange.com/questions/412580/… just R2 or "WMSE R2"? Jun 14, 2019 at 8:11
• Also one other question - would 1-DRES/DTOT be maximized by the original maximum likelihood objective of the GLM model fit? [For the measure I propose here, stats.stackexchange.com/questions/412580/…, I'm pretty sure that's the case] Jun 14, 2019 at 8:16
• @Dave: The specific form comes from substituting the density function for the normal distribution into the likelihood function. If you do that then the log-likelihood terms reduce to sums-of-squares.
– Ben
Jul 4, 2020 at 22:49
• @RobertLong: I was always taught that imitation is the sincerest form of flattery, so more than happy for the answer to be replicated elsewhere (with attribution). Your attribution looks fine to me, so all good. Thanks, Ben.
– Ben
May 22, 2021 at 2:48

Why should SSE + SSR be equal to SST? It just happened to be the case for the linear model. There are many ways to show that it should hold for $y=X\beta+\varepsilon$ under Gauss-Markov conditions. However, it doesn't need to hold in general case. The burden is to prove that it holds, not that it doesn't

• It should hold under orthogonality of $(Y_i-\hat{Y_i})$ and $(\hat{Y_i}-\bar{Y})$ (regression and residual) vectors. We can partition $\sum{(Y_i-\bar{Y})^2}$ into $\sum{((Y_i-\hat{Y_i})+(\hat{Y_i}-\bar{Y}))^2} = \sum{(Y_i-\hat{Y_i})^2} + \sum{(\hat{Y_i}-\bar{Y})^2} + 2*\sum{(Y_i-\hat{Y_i})*(\hat{Y_i}-\bar{Y})}$. If the two are orthogonal, then the third summation above should equal zero since it is the inner product of the vectors.
– Greg
Jul 31, 2018 at 3:24
• @Greg, orthogonality itself is a derived property, it's not a part of regression assumptions Jul 31, 2018 at 11:20
• I prefer to think of it in the 2-D case. Let's say you have vectors A and B in 2-D space. This is equivalent to SSTotal and SSR. SSE is the difference between SStotal and SSR, or (A - B). These three vectors form a triangle.
– Greg
Aug 6, 2018 at 23:09
• The projection is a linear concept Aug 6, 2018 at 23:23
• @Greg That last remark makes me wonder what you actually mean by "non-linear" regression, because it explicitly equates that with linear regression! I suspect that most readers have understood you to refer to models that are not linear in the parameters, whereas you seem to be referring to models that might be nonlinear in the original explanatory variables. But in that case, your assertion that the Pythagorean relationship does not hold is incorrect. Perhaps your question, then, amounts to confusing two different senses of "non-linear"?
– whuber
Aug 17, 2023 at 17:23

While R-squared may still be a flawed measurement in non-linear models for other reasons, I believe I have sufficiently shown that the SSR + SSE = SSTotal relationship still holds in a least-squares model for certain non-linear functions, especially those that allow for a constant term, such as polynomial models. I believe that this conclusion is compatible with what has been posted in this discussion, including what I read from the ncbi link provided, although I was unable to access the full report.

If one has a series of fitted values $\hat y_i$ with respect to a series of observations $y_i$, where $\hat y_i$ $= A + f(X) =$ $\bar Y$ $+ (A-\bar Y)$ $+ f(X)$, with $A$ being a constant term and $f(X)$ a function of predictor variables, in which the vector of $(\hat{Y_i} - \bar{Y})$ is not orthogonal to $(Y_i - \hat{Y_i})$, one can create a new set of fitted values $Z_i$ such that $Z_i = c*(\hat{Y_i} - \bar{Y}) + \bar{Y}$, where c = $\sum{(\hat{Y_i}-\bar{Y})*(Y_i-\hat{Y_i})} / \sum{(\hat{Y_i} - \bar{Y})^2}$. With new fitted values $Z_i$, the vector $(Z_i - \bar{Y})$ will be orthogonal to the error vector and this new error vector $(Y_i - Z_i)$ will have a smaller sum of squares than the original $(Y_i-\hat{Y_i})$. The $Z_i$ were simply obtained by multiplying the original estimated model by a constant $"c"$ and adding a multiple of the observations' mean, which is compatible with the model having a constant term. Therefore a least-squares model should always have orthogonal regression and error vectors in these circumstances, which means that $SSE + SSR = SSTotal$.

I have created polynomial models on a handful of datasets at work and this relationship has held with all of them. I am just saying.

• "least-squares model for certain non-linear functions, [...], such as polynomial models": If you meant models of the form y = \sum \beta_i P_i(x) where P_i(x) is a polynomial, then those are linear in the parameters \beta_i so no wonder the SS_R + SS_E = SS_T relationship holds :-) Aug 18, 2021 at 12:26

I think it's important to distinguish between

1. whether finite sample math as it relates to $$R^2$$, $$TSS$$, $$SSR$$, etc. works out in the non-linear case as it does with linear models;
2. whether $$R^2$$ is still informative in the non-linear, possibly misspecified, regression, case; and
3. to what extent is inference with an empirically determined $$R^2$$ rigorous.

Packages such as scikit-learn have implemented $$R^2$$ in their regression classes, so at least some people find it useful.

If

$$SSR=\sum_{i=1}^n(Y_i-\hat{Y}_i)^2$$

is not much smaller than

$$TSS = \sum_{i=1}^n(Y_i-\bar{Y})^2,$$

then my personal intuition tells me not to pick the regression method (whether linear or non-linear) that gave $$\hat{Y}_i$$. Depending on the method and amount of training data, I may even think that the predictors are not so good when $$SSR/TSS$$ is large.

But what is it that we are actually trying to infer with $$R^2=1-\frac{SSR}{TSS}?$$

If the approximation of $$\mathbb{E}[Y_i|X_i]$$, the conditional expectation (mean regression) function, by $$\hat{Y}_i$$ is good, it ought to be the case that

$$\frac{SSR}{n}$$ approximates well its population-level analouge $$\mathbb{E}[ (Y_i-\mathbb{E}[Y_i|X_i])^2 ] = \mathbb{E}[\text{Var}[Y_i|X_i]].$$

For this to hold, the bias of $$\hat{Y}_i$$ should be small.

Moreover, given large enough $$n$$, it should be the case that

$$\frac{{TSS}}{n}\approx\mathbb{E}[(Y_i-\mathbb{E}[Y_i])^2]=\text{Var}[Y_i].$$

If we believe this to be the case, then

$$R^2\approx 1-\frac{\mathbb{E}[ \text{Var}[Y_i|X_i] ] }{ \text{Var}[Y_i] }=:R^2_{ \text{population} }.$$

The law of total variance, which holds regardless of whether $$\mathbb{E}[Y_i|X_i]$$ is linear or not, helps us see that

$$R^2_{ \text{population} } = \frac{ \text{Var}[\mathbb{E}[Y_i|X_i]] }{ \text{Var}[\mathbb{E}[Y_i|X_i]]+\mathbb{E}[\text{Var}[Y_i|X_i]] }.$$

As I hope is becoming apparent, if

$$\frac{SSR}{n}$$ approximates $$\mathbb{E}[\text{Var}[Y_i|X_i]]$$

well, and $$R^2$$ is large, this seems to imply we can explain $$Y_i$$ well with $$X_i$$. This is because most of the variation in $$Y_i$$ comes from its relation to $$X_i$$.

Formal statistical inference of $$R^2_{\text{population}}$$ is most ideal. Alas, it is difficult - without assumptions - to rigorously characterize a general sampling distribution of $$R^2$$ as we can do in the linear regression case.

Assuming we are calculating $$R^2$$ with a fairly large test data set (or cross-validation sets), and given that we are sometimes not sure whether we've correctly specified our regression function, in practice it may be somewhat okay to assume the finite sample $$R^2$$ is an underestimate of the elusive population quantity $$R^2_{\text{population}}$$. Afterall, one can show that

$$\frac{SSR}{n}\approx\mathbb{E}[(Y_i-\hat{Y}_i)^2]\geq\mathbb{E}[(Y_i-\mathbb{E}[Y_i|X_i])^2].$$

$$R^2$$ is of limited use in nonlinear regression. We make it available in GraphPad Prism, but suggest it be used in only one way:

Look at $$R^2$$ when you run a series of experiments, and you want to make sure that today's experiment is consistent with other runs of the experiment. For example, if you always get $$R^2$$ between 0.90 and 0.95 but today you got $$R^2$$=0.75, then you should be suspicious and look carefully to see if something went wrong with the methods or reagents used in that particular experiment. And if a new employee brings you results showing $$R^2$$ of 0.99 using that same system, you should look carefully at how many "outliers" were removed, and whether some data were made up.

More.