Why is it important to make a distinction between "linear" versus "non-linear" regression?

Question

What is the importance of the distinction between linear and non-linear models? The question Nonlinear vs. generalized linear model: How do you refer to logistic, Poisson, etc. regression? and its answer was an extremely helpful clarification of the linearity/non-linearity of generalized linear models. It seems critically important to distinguish linear from non-linear models, but it is not clear to me why? For example, consider these regression models:

\begin{align} E[Y \mid X] & = \beta_0 + \beta_1 X \tag{1} \\ E[Y \mid X] & = \beta_0 + \beta_1 X + \beta_2 X^2 \tag{2} \\ E[Y \mid X] & = \beta_0 + \beta_1^2 X \tag{3} \\ E[Y \mid X] & = \{1+\exp(-[ \beta_0 + \beta_1 X]\}^{-1} \tag{4} \end{align}

Both Models 1 and 2 are linear, and the solutions to $\beta$ exist in closed form, easily found using a standard OLS estimator. Not so for Models 3 and 4, which are nonlinear because (some of) the derivatives of $E[Y\mid X]$ wrt $\beta$ are still functions of $\beta$.

One simple solution to estimate $\beta_1$ in Model 3 is to linearize the model by setting $\gamma = \beta_1^2$, estimate $\gamma$ using a linear model, and then compute $\beta_1 = \sqrt{\gamma}$.

To estimate the parameters in Model 4, we can assume $Y$ follows a binomial distribution (member of the exponential family), and, using the fact that the logistic form of the model is the canonical link, linearize the r.h.s. of the model. This was Nelder and Wedderburn's seminal contribution.

But why is this non-linearity a problem in the first place? Why can one not simply use some iterative algorithm to solve Model 3 without linearizing using the square root function, or Model 4 without invoking GLMs. I suspect that prior to widespread computational power, statisticians were trying to linearize everything. If true, then perhaps the "problems" introduced by nonlinearity are a remnant of the past? Are the complications introduced by non-linear models merely computational, or are there some other theoretical issues that make non-linear models more challenging to fit to data than linear models?

Answer 1

I can see two main differences:

• linearity makes it simple and robust. For instance, (linear) OLS is unbiased estimator under unknown disturbance distribution. In general, GLM and non-linear models are not. OLS is also robust for various error structure model (random effects, clustering, etc) where in non-linear models you typically have to assume the exact distribution of these terms.

• Solving it is easy: just a couple of matrix multiplications + 1 inverse. This means you can almost always solve it, even in cases where the objective function is almost flat (multicollinearity.) Iterative methods may not converge in such problematic cases (which, in a sense, is a good thing.) Easy solving may or may not be less of an issue nowadays. Computers get faster, but data get bigger. Ever tried to run a logit regression on 1G observations?

Besides of that, linear models are easier to interpret. In linear models marginal effects equal to coefficients and are independent of X values (although polynomial terms screw up this simplicity.)

Answer 2

Many models in biology (and other fields) are nonlinear, so are best fit with nonlinear regression. The math is very different, of course. But from the point of view of the data analyst, there really is only one important difference.

Nonlinear regression requires initial estimated values for each parameter. If these initial estimates are way off, the nonlinear regression program can converge on a false minimum and give useless or misleading results.

Answer 3

Firstly I am going to substitute the word 'model' for the word 'regression'. I think that for both words one is really asking what is the relevant equations that define the model and what are the relevant hypothesis relating the values of the dependent variable and the values predicted by the equation/model. I think that the term 'model' is more standard. If you agree with that, read on.

I really owe this answer to reflections on the comment of a colleague who is a classically trained probabilist and statistician. He objected violently to a book terming a polynomial regression as non-linear and that is when I read more seriously about non-linear models. I believe that the correct answer is that a linear model assumes that the error term is Gaussian whereas a generalized linear model assumes a more generalized form for the error term. If $\phi_1, \ldots, \phi_n$ are any set of functions, then one can attempt to construct a linear model in $\phi_1, \ldots, \phi_n$. For example if $\phi_i = x^i$ , then we get a polynomial regression. It is a linear model if the the difference $\epsilon_i = y_i - \sum a_{ij}x^j $ is Gaussian. Imho, I think wikipedia has a very reasonable explanation of general linear models. I think this is the key sentence - " The GLM generalizes linear regression by allowing the linear model to be related to the response variable via a link function and by allowing the magnitude of the variance of each measurement to be a function of its predicted value." So a glm allows a more general error term. This allows for greater flexibility in modeling. The price ? Calculating the correct model is harder. One no longer has a simple method of calculating the coefficients. The coefficients of a linear regression can be found by minimizing a quadratic functional which has a unique mimimum. In the words of Borat, for a glm , not so much. One has to calculate the mle, and further since the procedure is numerical there is definitely an issue with finding local minima versus actual minima.