Why is it important to make a distinction between "linear" versus "non-linear" regression?

What is the importance of the distinction between linear and non-linear models? The question Nonlinear vs. generalized linear model: How do you refer to logistic, Poisson, etc. regression? and its answer was an extremely helpful clarification of the linearity/non-linearity of generalized linear models. It seems critically important to distinguish linear from non-linear models, but it is not clear to me why? For example, consider these regression models:

\begin{align} E[Y \mid X] & = \beta_0 + \beta_1 X \tag{1} \\ E[Y \mid X] & = \beta_0 + \beta_1 X + \beta_2 X^2 \tag{2} \\ E[Y \mid X] & = \beta_0 + \beta_1^2 X \tag{3} \\ E[Y \mid X] & = \{1+\exp(-[ \beta_0 + \beta_1 X]\}^{-1} \tag{4} \end{align}

Both Models 1 and 2 are linear, and the solutions to $\beta$ exist in closed form, easily found using a standard OLS estimator. Not so for Models 3 and 4, which are nonlinear because (some of) the derivatives of $E[Y\mid X]$ wrt $\beta$ are still functions of $\beta$.

One simple solution to estimate $\beta_1$ in Model 3 is to linearize the model by setting $\gamma = \beta_1^2$, estimate $\gamma$ using a linear model, and then compute $\beta_1 = \sqrt{\gamma}$.

To estimate the parameters in Model 4, we can assume $Y$ follows a binomial distribution (member of the exponential family), and, using the fact that the logistic form of the model is the canonical link, linearize the r.h.s. of the model. This was Nelder and Wedderburn's seminal contribution.

But why is this non-linearity a problem in the first place? Why can one not simply use some iterative algorithm to solve Model 3 without linearizing using the square root function, or Model 4 without invoking GLMs. I suspect that prior to widespread computational power, statisticians were trying to linearize everything. If true, then perhaps the "problems" introduced by nonlinearity are a remnant of the past? Are the complications introduced by non-linear models merely computational, or are there some other theoretical issues that make non-linear models more challenging to fit to data than linear models?

• If you want to estimate $E[Y|X] = \beta_0 + \beta_1^2 X$, simply estimate $E[Y|X] = \beta_0 + \gamma X$ (simple linear regression) and then take $\beta_1 = \sqrt{\gamma}$...
– Tim
Dec 26, 2015 at 21:02
• @Tim, thanks for the comment. I was aware of this transformation as a possibility, but was trying to ask a somewhat different question. I have substantially edited the question, hopefully for the better. Dec 27, 2015 at 2:11
• I am unclear on the difference between your model 3 and $E[Y | X] = \beta_0 + \left(\sqrt{\beta_1}\right)^2 X$. Is your model 3 supposed to be constrained by a previous estimate of your model 1? Or did you mean to provide an example like $E[Y | X] = \beta_0 + \left(\beta_1\right)^{2X}$ or…? Jan 30, 2023 at 23:15

I can see two main differences:

• linearity makes it simple and robust. For instance, (linear) OLS is unbiased estimator under unknown disturbance distribution. In general, GLM and non-linear models are not. OLS is also robust for various error structure model (random effects, clustering, etc) where in non-linear models you typically have to assume the exact distribution of these terms.

• Solving it is easy: just a couple of matrix multiplications + 1 inverse. This means you can almost always solve it, even in cases where the objective function is almost flat (multicollinearity.) Iterative methods may not converge in such problematic cases (which, in a sense, is a good thing.) Easy solving may or may not be less of an issue nowadays. Computers get faster, but data get bigger. Ever tried to run a logit regression on 1G observations?

Besides of that, linear models are easier to interpret. In linear models marginal effects equal to coefficients and are independent of X values (although polynomial terms screw up this simplicity.)

• I the distinction as mainly one of convenience or historical usage. Dec 28, 2015 at 5:11

Many models in biology (and other fields) are nonlinear, so are best fit with nonlinear regression. The math is very different, of course. But from the point of view of the data analyst, there really is only one important difference.

Nonlinear regression requires initial estimated values for each parameter. If these initial estimates are way off, the nonlinear regression program can converge on a false minimum and give useless or misleading results.

• This certainly is part of the answer. But, by contending the only difference is something amounting to a minor technicality, you might be overly minimizing the problems of nonlinear models. For instance, some simple ones arising in biology can have sharply different local minima, all of which are close to global minima. This fundamental qualitative issue is not resolved by improved computing power or better optimization techniques: the very nature of many nonlinear models is so different from linear models that they require profound thought about their meaning and their interpretation.
– whuber
Dec 30, 2015 at 19:24

Firstly I am going to substitute the word 'model' for the word 'regression'. I think that for both words one is really asking what is the relevant equations that define the model and what are the relevant hypothesis relating the values of the dependent variable and the values predicted by the equation/model. I think that the term 'model' is more standard. If you agree with that, read on.

I really owe this answer to reflections on the comment of a colleague who is a classically trained probabilist and statistician. He objected violently to a book terming a polynomial regression as non-linear and that is when I read more seriously about non-linear models. I believe that the correct answer is that a linear model assumes that the error term is Gaussian whereas a generalized linear model assumes a more generalized form for the error term. If $\phi_1, \ldots, \phi_n$ are any set of functions, then one can attempt to construct a linear model in $\phi_1, \ldots, \phi_n$. For example if $\phi_i = x^i$ , then we get a polynomial regression. It is a linear model if the the difference $\epsilon_i = y_i - \sum a_{ij}x^j$ is Gaussian. Imho, I think wikipedia has a very reasonable explanation of general linear models. I think this is the key sentence - " The GLM generalizes linear regression by allowing the linear model to be related to the response variable via a link function and by allowing the magnitude of the variance of each measurement to be a function of its predicted value." So a glm allows a more general error term. This allows for greater flexibility in modeling. The price ? Calculating the correct model is harder. One no longer has a simple method of calculating the coefficients. The coefficients of a linear regression can be found by minimizing a quadratic functional which has a unique mimimum. In the words of Borat, for a glm , not so much. One has to calculate the mle, and further since the procedure is numerical there is definitely an issue with finding local minima versus actual minima.

• A nonlinear model can also assume that the residuals are sampled from a Gaussian distribution. A simple example is enzyme activity (Y) as a function of substrate concentration (X). Y=Vmax*X/(Km + X) It is common and sensible to assume that the residuals are gaussian, yet this is a nonlinear equation that is fit with nonlinear regression. Dec 31, 2015 at 19:04
• Nonlinear models comprise much more than GLMs. GLMs are popular because they are "almost" linear in the parameters: all the nonlinearity is confined to a function of a single variable, the "link." This allows for relatively efficient, reliable solutions. Other nonlinear models are much less tractable. The concept of linearity is largely separate from the nature of the residuals, although in some cases it is beneficial to distinguish additive residuals from other forms of variation.
– whuber
Dec 31, 2015 at 19:21