56
$\begingroup$

I have a (mixed) model in which one of my predictors should a priori only be quadratically related to the predictor (due to the experimental manipulation). Hence, I would like to add only the quadratic term to the model. Two things keep me from doing so:

  1. I think I read somehwere that you should always include the lower order polynomial when fitting higher order polynomials. I forgot where I found it and in the literature I looked at (e.g., Faraway, 2002; Fox, 2002) I cannot find a good explanation.
  2. When I add both, the linear and quadratic term, both are significant. When I add only one of them, they are not significant. However, a linear relation of predictor and data is not interpretable.

The context of my question is specifically a mixed-model using lme4, but I would like to get answers that could explain why it is or why it is not okay to inlcude a higher order polynomial and not the lower order polynomial.

If necessary I can provide the data.

$\endgroup$
  • 5
    $\begingroup$ I think the answers to this question might be helpful. $\endgroup$ – user10525 May 18 '12 at 13:36
  • 6
    $\begingroup$ Yes I agree with Procrastinator, and the interaction questions are essentially the same consideration. We have a few highly voted questions on the topic. In addition to Pro's suggestion, see also Do all interactions terms need their individual terms in regression model? and What if interaction wipes out my direct effects in regression?. $\endgroup$ – Andy W May 18 '12 at 13:41
  • $\begingroup$ Thanks for the reminder to these questions. From the answers given there it seems that it is an okay strategy if you have good a priori reasons to only include the quadratic term and not per se wrong. The question that remains is the one about scalability (see: stats.stackexchange.com/a/27726/442). Should I center my variable before fitting when only using the quadratic term? $\endgroup$ – Henrik May 18 '12 at 13:52
  • 1
    $\begingroup$ @Henrik - my answer in the link you posted was regarding how model inference in dependent on arbitrary shifts in the predictor values (such as mean centering) - it is undesirable to have substantive conclusion depends on something so arbitrary which is why my answer to your question is 'no', for the same reason. $\endgroup$ – Macro May 18 '12 at 16:09
  • 2
    $\begingroup$ The issue of quadratic vs linear is sufficiently conceptually distinct from interactions that I think this should not be considered a duplicate. $\endgroup$ – gung Jan 1 '17 at 23:48
65
$\begingroup$

1. Why include the linear term?

It is illuminating to notice that a quadratic relationship can be written in two ways:

$$y = a_0 + a_1 x + a_2 x^2 = a_2(x - b)^2 + c$$

(where, equating coefficients, we find $-2a_2 b = a_1$ and $a_2 b^2 + c = a_0$). The value $x=b$ corresponds to a global extremum of the relationship (geometrically, it locates the vertex of a parabola).

If you do not include the linear term $a_1 x$, the possibilities are reduced to

$$y = a_0 + a_2 x^2 = a_2(x - 0)^2 + c$$

(where now, obviously, $c = a_0$ and it is assumed the model contains a constant term $a_0$). That is, you force $b=0$.

In light of this, question #1 comes down to whether you are certain that the global extremum must occur at $x=0$. If you are, then you may safely omit the linear term $a_1 x$. Otherwise, you must include it.

2. How to understand changes in significance as terms are included or excluded?

This is discussed in great detail in a related thread at https://stats.stackexchange.com/a/28493.

In the present case, the significance of $a_2$ indicates there is curvature in the relationship and the significance of $a_1$ indicates that $b$ is nonzero: it sounds like you need to include both terms (as well as the constant, of course).

$\endgroup$
  • 1
    $\begingroup$ Thanks whuber. Great answer. So if I center the theoretical extremum on 0 (it is actually a minimum) I am fine with ommitting the linear term. This acutally leads to a highly significant quadratic predictor (without the linear one). $\endgroup$ – Henrik May 18 '12 at 16:10
  • $\begingroup$ if both linear and quadratic terms of a variable are correlated, can I include both of them in a model, or should I exclude one (which I assume it should be the quadratic)? $\endgroup$ – mtao Mar 3 '17 at 13:03
  • $\begingroup$ @Teresa There's no general reason to eliminate correlated terms in a regression. (If that were the case, the vast majority of regression models ever created would be in trouble!) Very strongly correlated terms that together contribute nothing meaningful to the model's fit compared to either term alone can be reduced to a subset of those terms. $\endgroup$ – whuber Mar 3 '17 at 23:32
  • $\begingroup$ @whuber, thank you very much! Also, for a logistic regression model, I used odds ratio to estimate the effect size, but only with linear terms. When I have linear and quadratic, can I use the same approach and interpretate the results in the same way? $\endgroup$ – mtao Mar 4 '17 at 18:24
  • $\begingroup$ Not quite. The reason is that you cannot separately change the linear and quadratic terms. You have to consider how the response will change when you change the original variable a little bit. $\endgroup$ – whuber Mar 4 '17 at 18:31
22
$\begingroup$

@whuber has given a really excellent answer here. I just want to add a small complimentary point. The question states that "a linear relation of predictor and data is not interpretable". This hints at a common misunderstanding, although I usually hear it on the other end ('what is the interpretation of the squared [cubic, etc.] term?').

When we have a model with multiple different covariates, each beta [term] can generally be afforded its own interpretation. For example, if:
$$ \widehat{\text{GPA}}_{college}=\beta_0+\beta_1\text{GPA}_{highschool}+\beta_2\text{class rank}+\beta_3\text{SAT}, $$

(GPA means grade point average;
rank is the ordering of a student's GPA relative to other students at the same high school; &
SAT means 'scholastic aptitude test' a standard, nationwide test for students going to university)

then we can assign separate interpretations to each beta/term. For instance, if a student's high school GPA were 1 point higher--all else being equal--we would expect their college GPA to be $\beta_1$ points higher.

It is important to note, however, that it is not always permissible to interpret a model in this manner. One obvious case is when there is an interaction amongst some of the variables, as it would not be possible for the individual term to differ and still have all else held constant--of necessity, the interaction term would change as well. Thus, when there is an interaction, we do not interpret main effects but only simple effects, as is well understood.

The situation with power terms is directly analogous, but unfortunately, does not seem to be widely understood. Consider the following model:
$$ \hat{y}=\beta_0+\beta_1x+\beta_2x^2 $$ (In this situation, $x$ is intended to represent a prototypical continuous covariate.) It is not possible for $x$ to change without $x^2$ changing also, and vice versa. Simply put, when there are polynomial terms in a model, the various terms based on the same underlying covariate are not afforded separate interpretations. The $x^2$ ($x$, $x^{17}$, etc.) term does not have any independent meaning. The fact that a $p$-power polynomial term is 'significant' in a model indicates that there are $p-1$ 'bends' in the function relating $x$ and $y$. It is unfortunate, but unavoidable, that when curvature exists, the interpretation becomes more complicated, and possibly less intuitive. To assess the change in $\hat{y}$ as $x$ changes, we will have to use calculus. The derivative of the above model is:
$$ \frac{dy}{dx}=\beta_1+2\beta_2x $$ which is the instantaneous rate of change in the expected value of $y$ as $x$ changes, all else being equal. This is not so clean as the interpretation of the very top model; importantly, the instantaneous rate of change in $y$ depends on the level of $x$ from which the change is assessed. Furthermore, the rate of change in $y$ is an instantaneous rate; that is, it is itself continuously changing throughout the interval from $x_{old}$ to $x_{new}$. This is simply the nature of a curvilinear relationship.

$\endgroup$
  • 1
    $\begingroup$ Excellent response! This reminds me a few excellent responses user chl has provided on interpreting interaction effects. He gives article references in this response,What are best practices in identifying interaction effects?. And gives a wonderful example of graphically displaying interaction using coplots in this response, Is interaction possible between two continuous variables?. $\endgroup$ – Andy W May 18 '12 at 17:58
  • 1
    $\begingroup$ To Gung's answer I just want to say that statistical modeling involves noise which can disguise details in a polynomial regression model. i think that the centering issue that Bill Huber raised was a greta one because in one formualtion a linear term is missing and in the other it occurs with the quadratic term. The strength of the curvature in the signal dictates the need for a higher than first order term but really tells us nothing about the need for a linear term also. $\endgroup$ – Michael Chernick May 18 '12 at 23:58
7
$\begingroup$

@whuber's answer above is right on target in pointing out that omitting the linear term is the "usual" quadratic model is equivalent to saying, "I am absolutely certain that the extremum is at $x=0$."

However, you also need to check whether the software you are using has a "gotcha". Some software may automatically center the data when fitting a polynomial and testing its coefficients unless you turn off polynomial centering. That is, it may fit an equation that looks something like $Y = b_0 + b_2(x - \bar{x})^2$ where $\bar{x}$ is the mean of your $x$s. That would force the extremum to be at $x=\bar{x}$.

Your statement that both the linear and quadratic terms are significant when both are entered needs some clarification. For example, SAS may report a Type I and/or a Type III test for that example. Type I tests the linear before putting in the quadratic. Type III tests the linear with the quadratic in the model.

$\endgroup$
  • 2
    $\begingroup$ This is a reasonable point, but just b/c the data were centered prior to creating $x^2$ doesn't mean you can be "absolutely certain that the extremum is at $x=0$". Saying that now is equivalent to having said "the extremum is at $x=\bar{x}$" before. In either case you are betting the unbiasedness of your model on your ability to specify the x-value of the extremum w/ infinite precision. The difference b/t Type I & Type III tests is also a potentially interesting addition, but nb, they would only differ if $x$ & $x^2$ are correlated, ie, if centering had not occurred. $\endgroup$ – gung May 22 '12 at 21:22
  • $\begingroup$ On a different note, you can refer to a user's contributions by stating their username, possible with the 'at' symbol. Eg, in this case, '@whuber's answer is right on target...' (A sentiment with which I agree.) $\endgroup$ – gung May 22 '12 at 21:24
  • 1
    $\begingroup$ Thank you, Emil, for contributing those reminders: they are both worth bearing in mind. $\endgroup$ – whuber May 22 '12 at 23:05
3
$\begingroup$

Brambor, Clark and Golder (2006) (which comes with an internet appendix) have a very clear take on how to understand interaction models and how to avoid the common pitfalls, including why you should (almost) always include the lower-order terms ("constitutive terms") in interaction models.

Analysts should include all constitutive terms when specifying multiplicative interaction models except in very rare circumstances. By constitutive terms, we mean each of the elements that constitute the interaction term. [..]

The reader should note, though, that multiplicative interaction models can take a variety of forms and may involve quadratic terms such as $X^2$ or higher-order interaction terms such as $XZJ$. No matter what form the interaction term takes, all constitutive terms should be included. Thus, $X$ should be included when the interaction term is $X^2$ and $X$, $Z$, $J$, $XZ$, $XJ$, and $ZJ$ should be included when the interaction term is $XZJ$.

Failure to do so may result in an underspecified model that would lead to biased estimates. This may lead to inferential errors.

If this is the case and $Z$ is correlated with either $XZ$ (or $X$) as will occur in virtually any social science circumstance, then omitting the constitutive term $Z$ will result in biased (and inconsistent) estimates of $\beta_0$, $\beta_1$, and $\beta_3$. Although not always recognized as such, this is a straightforward case of omitted variable bias (Greene 2003, pp. 148–149).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.