Say I have outcome variable $Y_i$ and predictors $X_{i1}$ and $X_{i2}$ for some data point $i$. Wikipedia says that a model is linear when:

the mean of the response variable is a linear combination of the parameters (regression coefficients) and the predictor variables.

I thought this meant that a model can be no more complicated than: $Y_i = \beta_1 X_{i1} + \beta_2 X_{i2}$. However, upon further reading, I found out you could handle non-linear "interactions" of the predictors like in $Y_i = \beta_1 X_{i1} + \beta_2 X_{i2} + \beta_3 X_{i1}\ X_{i2}$ by viewing $X_{i1}\ X_{i2}$ as just another predictor (which happens to be dependent on $X_{i1}$ and $X_{i2}$). This seems to mean that you can use any (linear or non-linear) function of the predictors like $\log(X_{i1} / X_{i2}^2)$ or whatever. Conceptually, this type of "recoding" seems like it should work for the coefficients as well.

So: What exactly are the limits of linear regression, given you can do this kind of manipulation?


The parameter needs to enter linearly into the equation. So something like $E(Y)=\beta_1 \cos(\beta_2 x_i + \beta_3)$ would not qualify. But you can take functions of the independent variables as follows:

$E(Y)=\beta_0 + \beta_1X_i + \beta_2X^2 + \beta_3 e^{X_i}$

for example.

So the limits of linear regressions are: the mean of the $Y$ values is of the form parameter times (independent variable stuff) + parameter times (more independent variable stuff) ... and so on.

| cite | improve this answer | |
  • $\begingroup$ Sorry for my math-ineptitude but what is the official definition of "entering linearly" into an equation? $\endgroup$ – user3243135 May 31 '14 at 4:53
  • $\begingroup$ user2243865: The parameters enter linearly when they see it as a linear function. In the second equation, that's the case, but it isn't in the first equation. $\endgroup$ – Glen_b May 31 '14 at 7:38
  • $\begingroup$ Ah, ok, I was confused -- linear means it only needs to be "linear in" the coefficients. $\endgroup$ – user3243135 May 31 '14 at 8:17
  • $\begingroup$ user2243865: It's also "linear in the entered predictors". The term $\beta_3e^{x}$ is not linear in $x$, but it is linear in the thing $\beta_3$ is the coefficient for ($e^{x}$), which is what gets supplied as the fourth column of the X-matrix. In matrix notation, $E(Y)=X\beta$ is linear in both $\beta$ and $X$.... it's just that the columns of $X$ aren't necessarily linear in some variable, $x$, which is how linear models can fit some forms of nonlinear relationships. $\endgroup$ – Glen_b Jun 4 '14 at 0:55
  • $\begingroup$ Placidia: neither of your equations can be true if there's any noise (/error) in your observations (without which why would we need regression at all?). You either need an error term on the RHS or to put an expectation on the LHS. $\endgroup$ – Glen_b Jun 4 '14 at 0:58

(Almost) Everything can be expressed as a linear model, if you don't restrict it to a finite number of parameters.

This is the basis of functional analysis and kernel regression (as in SVMs with kernels). For instance, Fourier series - you can produce an infinite sine/cosine series, where the amplitude of the wave of each frequency gets a learned coefficient, and you can learn (almost) any function (any function whose square is integrable - which is a very weak condition).

Kernel machines, and functional analysis, are a wonderful idea, and make the world seem very beautiful - virtually everything is linear!

See http://en.wikipedia.org/wiki/Kernel_methods

The classic statistical probabilistic reference is Grace Wahba's Spline Models for Observational Data.

| cite | improve this answer | |
  • $\begingroup$ I've only heard of kernels in random and seemingly unrelated places, like for kernel density estimation and convolutions in general. Do you have any recommendations for literature from a more mathematical/probabilistic point-of-view? (as opposed to algorithm/computer science perspectives, which I have a harder time digesting). $\endgroup$ – user3243135 May 31 '14 at 7:24
  • 2
    $\begingroup$ I upvoted this for the sheer boldness of the assertion. Mathematics is full of approximation theorems of this sort. But the fact remains that non-linearity is real, with distinct characteristics of its own. An approximation is only good for as far as it goes. And as a practical matter, I do like to limit my models to a finite number of parameters. $\endgroup$ – Placidia Jun 1 '14 at 3:28
  • $\begingroup$ user2243865: I updated the post with your reference $\endgroup$ – Joe Jun 1 '14 at 19:53
  • $\begingroup$ Placidia: Actually, Fourier and functional analysis are not approximation results, but show that wide classes of functions (i.e. square integrable) can be represented exactly by infinite linear models. Approximation comes in where you can regularization/take the first n terms of the series, etc., and come close, so that you have a finite model with low approximation error. $\endgroup$ – Joe Jun 1 '14 at 19:54
  • 2
    $\begingroup$ @Joe, I get that. But I'm old fashioned, and when I studied pure mathematics, back the day, infinite series were always addressed as the limits of finite series. So concretely, you are always dealing with a finite series. But even if you could take an infinite series, typically those convergence results only apply on a compact set. Outside that region, the approximand could differ a lot ielike feedback loops, tipping points and singularites, are essentially non-linear behaviours, which perhaps deserve non-linear models. $\endgroup$ – Placidia Jun 2 '14 at 0:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.