Interpreting nonlinear regression $R^2$

Question

In ordinary least squares linear regression, $R^2=1-\frac{SSRes}{SSTotal}$ is described as the “proportion of variance explained”. Does this apply to nonlinear regression, too?

Answer 1

NO

Let’s start by deriving $R^2$ in the linear case.

Notation

$y_i$ is observation $i$ of some response variable $Y$.

$\hat{y}_i$ is the value of $y_i$ predicted by the regression.

$\bar{y}$ is the average of all observations of the response variable.

$$ y_i-\bar{y} = (y_i - \hat{y_i} + \hat{y_i} - \bar{y}) = (y_i - \hat{y_i}) + (\hat{y_i} - \bar{y}) $$

$$( y_i-\bar{y})^2 = \Big[ (y_i - \hat{y_i}) + (\hat{y_i} - \bar{y}) \Big]^2 = (y_i - \hat{y_i})^2 + (\hat{y_i} - \bar{y})^2 + 2(y_i - \hat{y_i})(\hat{y_i} - \bar{y}) $$

$$SSTotal := \sum_i ( y_i-\bar{y})^2 = \sum_i(y_i - \hat{y_i})^2 + \sum_i(\hat{y_i} - \bar{y})^2 + 2\sum_i\Big[ (y_i - \hat{y_i})(\hat{y_i} - \bar{y}) \Big]$$

$$ :=SSRes + SSReg + Other $$

Divide through by the sample size $n$ (or $n-1$) to get variance estimates.

In OLS linear regression, $Other$ drops to zero. Consequently, all of the variance in $Y$ is accounted for by the residual variance (unexplained) and regression variance (explained). We, therefore, can describe the proportion of total variance explained by the regression, which would be the variance explained by the regression model $(SSReg/n)$ divided by the total variance $(SSTotal/n)$.

$$ \dfrac{SSReg/n}{SSTotal/n} $$$$= \dfrac{SSReg}{SSTotal} $$$$= \dfrac{SSTotal -SSRes-Other}{SSTotal} $$$$= 1-\dfrac{SSRes}{SSTotal} $$

That final line is a common definition of $R^2$ (and equivalent to other common definitions like squared correlation in the two-variable setting, and squared correlation between predictions and true $y$ values in a multiple linear regression that has several predictor variables (assuming an intercept parameter estimate)).

However, that relied on $Other =0$. When that is false, as it is in nonlinear regression, the formula is not so clean. There’s something contributing to the total variance besides the residual and regression variances, and the usual $R^2$ no longer means what it meant in OLS linear regression.

This does not invalidate $R^2$ as a performance metric in nonlinear regression, however. Aside from possible numerical funkiness that comes from doing math on a computer, minimizing mean squared error (MSE), which is common in regression, is equivalent to minimizing $SSRes$ or maximizing $R^2$, so if you were comfortable using MSE, you should be comfortable using $R^2$.

However, since $Other\ne 0$, it would be incorrect to interpret $R^2=1-\dfrac{SSRes}{SSTotal}$ as the proportion of variance explained.

(Why we don’t seek to maximize $SSReg$ instead of minimizing $SSRes$ is the subject of another question by someone with a username that might look familiar, and I do believe the question here to be somewhat different.)