14
$\begingroup$

I am an MBA student that is taking courses in statistics. Yesterday, I attended a statistics seminar in which some graduate students presented their research on some psychology experiments (e.g. response of mice to some stimulus).

The students presented some theory behind the models they used - in particular, they showed us about "General Linear Models" (GLM). I think I was able to follow the general ideas that were discussed. It seems to me that GLM's model some function of the mean instead of directly modelling the mean - this gives GLM's the advantage of being more flexible and versatile in modelling more complicated datasets.

  • The one thing that I couldn't understand however, is that why is the "Linear" part of GLM's so important? Why can't you just use/call it "General Models"?

  • When I tried to raise this point in the question and answer period, someone told me that the "Linear" aspect of GLM's allow for "ease in estimation" (i.e. easier "number crunching") compared to "Non-Linear" models and that the "Linear" aspect also allows for easier interpretation of the model compared to "Non-Linear" Models.

I am also not sure what exactly is a "Non-Linear" Model. I would have thought that a Logistic Regression is a "Non-Linear" Model because the output looks non-linear and it can model non-linear relationships, but it was explained to me that any model that can be written "linearly" like "x1b1 + x2b2 + ..." is called a "Linear Model". Therefore, is something like a Deep Neural Network a "Non-Linear Model"?

I couldn't really understand why this is true - can someone please help me with this?

$\endgroup$
3
  • $\begingroup$ "Linear" aspect is because they are linear in their parameters. $\endgroup$ Commented Sep 17, 2022 at 20:31
  • 1
    $\begingroup$ "In spite of the availability of highly innovative tools in statistics, the main tool of the applied statistician remains the linear model. The linear model involves the simplest & seemingly most restrictive statistical properties: independence, normality, constancy of variance, and linearity. However, the model and the statistical methods associated with it are surprisingly versatile and robust. More importantly, mastery of the linear model is a prerequisite to work with advanced statistical tools because most advanced tools are generalizations of the linear model." - Alvin C. Rencher. $\endgroup$ Commented Sep 17, 2022 at 20:51
  • 1
    $\begingroup$ Fun fact: I'm chemometrician, and most chemists would call a model/function linear if it is linear in the data (similar to OP). As others have said, in stats, the important linearity is in the parameters. To avoid ambiguity, in chemometrics we speak about bilinear models, which are both linear in data and in the parameters :-). (And they are of huge importance to us, because we often know from the physics underlying the data generation processes that bilinearity is at least a good approximation) $\endgroup$
    – cbeleites
    Commented Sep 18, 2022 at 12:28

2 Answers 2

15
$\begingroup$
$\endgroup$
10
  • 6
    $\begingroup$ I would temper "remarkably well" a bit. Linear models can be ruined by outliers and by not already knowing a good transformation for Y on which to base the analysis. $\endgroup$ Commented Sep 17, 2022 at 22:37
  • $\begingroup$ Thank you so much for addressing these points! Just two questions I had: $\endgroup$
    – stats_noob
    Commented Sep 18, 2022 at 2:39
  • $\begingroup$ 1) What is an example of a non-linear model? $\endgroup$
    – stats_noob
    Commented Sep 18, 2022 at 2:39
  • 1
    $\begingroup$ @MBA_Grad_Student_2022 when response variable is related with the predictor variables through non-linear function, say, $y = \theta_1\exp(\theta_2x) +\varepsilon.$ The normal equations corresponding to non-linear least squares aren't that easy to solve compared to the linear counterparts. $\endgroup$ Commented Sep 18, 2022 at 5:32
  • 2
    $\begingroup$ 2) Usually yes. But not always. For example, a decision tree is non-linear and easy to fit. $\endgroup$
    – Tim
    Commented Sep 18, 2022 at 5:34
6
$\begingroup$

The reason that linear models are so common in statistics is that they are simple and they allow relatively simple derivation of results. These models are more general and flexible than you might imagine, since "linearity" in this context means linearity with respect to the model parameters, not linearity with respect to the explanatory variables.

In regression analysis we are seeking to infer the conditional distribution of a response variable $Y$ conditional on an explanatory vector $\mathbf{x}$. This generally means that we make an inference about the conditional expected value of the response (called the "true regression function") and we also make an inference about the conditional distribution/variability around this conditional expectation. The linearity property in a GLM is a property pertaining to the true regression function; it means that you can write the main model equation for this function in this form:

$$g(\mathbb{E}(Y|\mathbf{x}, \boldsymbol{\beta})) = \beta_0 + \sum_{i=1}^k \beta_i f_i(\mathbf{x}),$$

where $g$ is the link function in the model and $f_1,...,f_k$ are a set of functions that transform the explanatory vector $\mathbf{x}$. Observe here that linearity is with respect to the parameter vector $\boldsymbol{\beta}$, not the explanatory vector $\mathbf{x}$ (though in some cases the equation is also an affine function of the latter). Below I give some examples of linear and nonlinear regression equations, to give you a flavour of the breadth of the linear model and the types of cases that fall outside its scope.

In order to understand the "simplicity" of the linear model, you will need to learn the derivation of inference results in GLMs and other linear models. If you take some time to learn how to derive the inference results from the underlying model form you will see that there are a number of steps in the derivation that use the linear form of the model, and which become much more complicated when dealing with a nonlinear model. It is possible to derive inference results and other properties for nonlinear models, but the derivation is substantially more complicated and it often requires numerical methods where nonlinear functions are locally approximated by linear functions. If you would like to learn this in depth, I recommend you devote some time to learning the derivation of the inference results in linear regression and GLMs; there is really no substitute here for actually getting into the first-principles derivations of the inference results in the linear model to see how the linear form makes things easier.


Examples of linear and nonlinear model equations: The following equations all give linear models:

$$\begin{align} \mathbb{E}(Y|\mathbf{x}, \boldsymbol{\beta}) &= \beta_0 + \sum_{i=1}^k \beta_i x_i, \\[6pt] \mathbb{E}(Y|x, \boldsymbol{\beta}) &= \beta_0 + \sum_{i=1}^k \beta_i x^i, \\[6pt] \mathbb{E}(Y|\mathbf{x}, \boldsymbol{\alpha}) &= \alpha_0 \prod_{i=1}^k \alpha_i^{x_i}, \\[6pt] \mathbb{E}(Y|\mathbf{x}, \boldsymbol{\alpha}) &= \alpha_0 \prod_{i=1}^k x_i^{\alpha_i}, \\[14pt] \mathbb{E}(Y|x, \alpha, \kappa) &= \alpha \sin ( 2 \pi (x + \kappa) ). \\[14pt] \end{align}$$

(As an exercise, see if you can successfully write each of these model equations in explicitly linear form.) The following equations all give nonlinear models:

$$\begin{align} \mathbb{E}(Y|\mathbf{x}, \boldsymbol{\alpha}) &= \alpha_0 + \prod_{i=1}^k \alpha_i^{x_i}, \\[6pt] \mathbb{E}(Y|\mathbf{x}, \boldsymbol{\beta}) &= \beta_0 + \sum_{i=1}^k e^{\beta_i x_i}, \\[14pt] \mathbb{E}(Y|x, \boldsymbol{\alpha}) &= \alpha_0 x e^{-\alpha_1 x}, \\[20pt] \mathbb{E}(Y|x, \alpha, \kappa) &= e^{-\alpha x} \sin ( 2 \pi (x + \kappa) ), \\[20pt] \mathbb{E}(Y|x, \alpha, \phi, \kappa) &= \alpha \sin ( 2 \pi \phi (x + \kappa) ). \\[14pt] \end{align}$$

(As an exercise, attempt to put these model equations into explicit linear form, and see why you can't do it.) It is also worth noting that the nonlinear equations can be locally approximated by linear equations using either Taylor approximation or Fourier approximation, so while the latter are not linear, they can be approximated by linear equations reasonably well in some cases.

$\endgroup$
2
  • $\begingroup$ @ Ben: thank you! can you please explain why the fourth and the fifth example are linear models? thank you so much! $\endgroup$
    – stats_noob
    Commented Oct 2, 2022 at 2:37
  • $\begingroup$ @stats_noob: Hint: For the fourth, try taking logarithms of both sides; for the fifth, see if you can find a trigonometry rule that lets you split an out-of-phase sinusoid into a linear combination of two simpler sinusoids. $\endgroup$
    – Ben
    Commented Sep 4 at 6:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.