4
$\begingroup$

I have a multivariate linear model (y=x1+x2) which gives me the following results when using R's plot() function: enter image description here

I can clearly see that the Normality and Linearity assumptions are not the best. Thus, I decided to add a 2nd degree polynomial variable to the model using poly() (which should give me a model such as y=x1^2+x1+x2) and got the following results: enter image description here

I clearly see an improvement in both assumptions, however, that made me think: can I even do these two tests after I included the polynomial variable? Is the model still considered linear?

$\endgroup$
7
  • 1
    $\begingroup$ I am not well versed in R's specifics w.r.t. poly(), but assuming it adds a term that is polynomial in the data, you should be fine. A linear model is linear in the parameters, not necessarily in the data. $\endgroup$ Jun 20 '18 at 8:51
  • 1
    $\begingroup$ poly() is used on the model itself, e.i: y~poly(x1,2)+x2. The data stays untouched. $\endgroup$ Jun 20 '18 at 9:29
  • 3
    $\begingroup$ Yes, the model is still linear (in its parameters and that's what is important here). Yes, you can use these plots. $\endgroup$
    – Roland
    Jun 20 '18 at 10:57
  • 1
    $\begingroup$ The term "linear" here has two meanings. One is that the model is a straight line. The other is that linear algebra can directly be used to solve for the coefficients in a technique named "linear regression". One is the name of a mathematical model, one is the name of a mathematical technique. $\endgroup$ Jun 20 '18 at 13:09
  • 2
    $\begingroup$ A linear regression model is defined as $\mathbf{y} = X\boldsymbol\beta + \boldsymbol\varepsilon$ where $X$ is the design matrix (all your x-variables) and $\boldsymbol\beta$ is the parameter vector. In the design matrix, a column can be squared values of another column, it won't change this basic, linear model. (Note that poly by default creates orthogonal polynomials.) $\endgroup$
    – Roland
    Jun 20 '18 at 13:22
2
$\begingroup$

Is the model still considered linear

To answer your question, recall the definition of a linear model.

Given a dataset composed of a vector $\mathbf{x} = \{ x_1, x_2,...,x_n\}$ of $n$ explanatory variables and one dependent variable $y$ we assume in this model that the relationship between $\mathbf{x}$ and $y$ is linear

$$ y = \beta_0 1 + \beta_1 x_1 + \beta_2 x_2 + ... + \beta_n x_n + \epsilon$$

Where $\beta_0$ is an intercept term and $\epsilon$ is the error variable, an unobserved random variable that adds "noise" to the linear relationship.

As @Roland points out, the linear relationship must hold in the parameters, not necessarily in the data. So nothing stops you from taking functions of the explanatory variables, and then performing the linear regression again. For example in your case, you could let:

$$ z_1 = x_1, \ z_2 = x_2, \ z_3 = x_1^2, \ z_4 = x_2^2$$

And then perform linear regression on $z$ as:

$$ y = \beta_0 1 + \beta_1 z_1 + \beta_2 z_2 + \beta_3 z_3 + \beta_4 z_4 + \epsilon $$ which is equivalent to

$$ y = \beta_0 1 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_1^2 + \beta_4 x_2^2 + \epsilon$$

I clearly see an improvement in both assumptions

If your data shows signs of an underlying polynomial relationship between the explanatory variables, then fitting a linear regression model on polynomial variables will improve your model. As always, this comes with many advantages and disadvantages, some discussed in the posts linked below.

An example

Here is a toy example of trying to fit a linear regression model on a noisy sine curve. As you may know, the sine curve can be approximated by a sum of polynomials, so intuitively we would expect a polynomial linear regression model to do well under certain conditions:

enter image description here

More details

See these excellent posts for more details and explanations

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.