Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.

I have a (mixed) model in which one of my predictors should a priori only be quadratically related to the predictor (due to the experimental manipulation). Hence, I would like to add only the quadratic term to the model. Two things keep me from doing so:

  1. I think I read somehwere that you should always include the lower order polynomial when fitting higher order polynomials. I forgot where I found it and in the literature I looked at (e.g., Faraway, 2002; Fox, 2002) I cannot find a good explanation.
  2. When I add both, the linear and quadratic term, both are significant. When I add only one of them, they are not significant. However, a linear relation of predictor and data is not interpretable.

The context of my question is specifically a mixed-model using lme4, but I would like to get answers that could explain why it is or why it is not okay to inlcude a higher order polynomial and not the lower order polynomial.

If necessary I can provide the data.

share|improve this question
4  
I think the answers to this question might be helpful. – user10525 May 18 '12 at 13:36
3  
Yes I agree with Procrastinator, and the interaction questions are essentially the same consideration. We have a few highly voted questions on the topic. In addition to Pro's suggestion, see also Do all interactions terms need their individual terms in regression model? and What if interaction wipes out my direct effects in regression?. – Andy W May 18 '12 at 13:41
Thanks for the reminder to these questions. From the answers given there it seems that it is an okay strategy if you have good a priori reasons to only include the quadratic term and not per se wrong. The question that remains is the one about scalability (see: stats.stackexchange.com/a/27726/442). Should I center my variable before fitting when only using the quadratic term? – Henrik May 18 '12 at 13:52
1  
@Henrik - my answer in the link you posted was regarding how model inference in dependent on arbitrary shifts in the predictor values (such as mean centering) - it is undesirable to have substantive conclusion depends on something so arbitrary which is why my answer to your question is 'no', for the same reason. – Macro May 18 '12 at 16:09

3 Answers

up vote 29 down vote accepted

1. Why include the linear term?

It is illuminating to notice that a quadratic relationship can be written in two ways:

$$y = a_0 + a_1 x + a_2 x^2 = a_2(x - b)^2 + c$$

(where, equating coefficients, we find $-2a_2 b = a_1$ and $a_2 b^2 + c = a_0$). The value $x=b$ corresponds to a global extremum of the relationship (geometrically, it locates the vertex of a parabola).

If you do not include the linear term $a_1 x$, the possibilities are reduced to

$$y = a_0 + a_2 x^2 = a_2(x - 0)^2 + c$$

(where now, obviously, $c = a_0$ and it is assumed the model contains a constant term $a_0$). That is, you force $b=0$.

In light of this, question #1 comes down to whether you are certain that the global extremum must occur at $x=0$. If you are, then you may safely omit the linear term $a_1 x$. Otherwise, you must include it.

2. How to understand changes in significance as terms are included or excluded?

This is discussed in great detail in a related thread at http://stats.stackexchange.com/a/28493.

In the present case, the significance of $a_2$ indicates there is curvature in the relationship and the significance of $a_1$ indicates that $b$ is nonzero: it sounds like you need to include both terms (as well as the constant, of course).

share|improve this answer
1  
Thanks whuber. Great answer. So if I center the theoretical extremum on 0 (it is actually a minimum) I am fine with ommitting the linear term. This acutally leads to a highly significant quadratic predictor (without the linear one). – Henrik May 18 '12 at 16:10

@whuber has given a really excellent answer here. I just want to add a small complimentary point. The question states that "a linear relation of predictor and data is not interpretable". This hints at a common misunderstanding, although I usually hear it on the other end ('what is the interpretation of the squared [cubic, etc.] term?').

When we have a model with multiple different covariates, each beta [term] can generally be afforded its own interpretation. For example, if:
$$ \widehat{\text{GPA}}_{college}=\beta_0+\beta_1\text{GPA}_{highschool}+\beta_2\text{class rank}+\beta_3\text{SAT}, $$

(GPA means grade point average;
rank is the ordering of a student's GPA relative to other students at the same high school; &
SAT means 'scholastic aptitude test' a standard, nationwide test for students going to university)

then we can assign separate interpretations to each beta/term. For instance, if a student's high school GPA were 1 point higher--all else being equal--we would expect their college GPA to be $\beta_1$ points higher.

It is important to note, however, that it is not always permissible to interpret a model in this manner. One obvious case is when there is an interaction amongst some of the variables, as it would not be possible for the individual term to differ and still have all else held constant--of necessity, the interaction term would change as well. Thus, when there is an interaction, we do not interpret main effects but only simple effects, as is well understood.

The situation with power terms is directly analogous, but unfortunately, does not seem to be widely understood. Consider the following model:
$$ \hat{y}=\beta_0+\beta_1x+\beta_2x^2 $$ (In this situation, $x$ is intended to represent a prototypical continuous covariate.) It is not possible for $x$ to change without $x^2$ changing also, and vice versa. Simply put, when there are polynomial terms in a model, the various terms based on the same underlying covariate are not afforded separate interpretations. The $x^2$ ($x$, $x^{17}$, etc.) term does not have any independent meaning. The fact that a $p$-power polynomial term is 'significant' in a model indicates that there are $p-1$ 'bends' in the function relating $x$ and $y$. It is unfortunate, but unavoidable, that when curvature exists, the interpretation becomes more complicated, and possibly less intuitive. To assess the change in $\hat{y}$ as $x$ changes, we will have to use calculus. The derivative of the above model is:
$$ \frac{dy}{dx}=\beta_1+2\beta_2x $$ which is the instantaneous rate of change in the expected value of $y$ as $x$ changes, all else being equal. This is not so clean as the interpretation of the very top model; importantly, the instantaneous rate of change in $y$ depends on the level of $x$ from which the change is assessed. Furthermore, the rate of change in $y$ is an instantaneous rate; that is, it is itself continuously changing throughout the interval from $x_{old}$ to $x_{new}$. This is simply the nature of a curvilinear relationship.

share|improve this answer
I like these thoughtful comments, but would you mind revisiting the last section? (Take the derivative or difference quotient of $\beta_1 x + \beta_2 x^2$ wrt $x$ to see where the error is.) Also, I suspect you might be inclined to modify your analysis--and conclusions--if you were to contemplate the possibility that $x$ may be categorical. You might also want to consider the possibility of using a set of orthogonal polynomials as a basis rather than the monomials: now, arguably, the terms do have "independent meanings." – whuber May 18 '12 at 17:06
@whuber, thanks for pointing that out. I had hoped to avoid using calculus & stay w/ conceptually-easier high school algebra. Clearly, that was an embarrassingly poor choice. Let me know if this remains inaccurate. Re: a categorical covariate, by default I think in terms of reference cell coding, thus, I have no idea what it would mean to have an $x^2$ term. – gung May 18 '12 at 17:40
1  
Excellent response! This reminds me a few excellent responses user chl has provided on interpreting interaction effects. He gives article references in this response,What are best practices in identifying interaction effects?. And gives a wonderful example of graphically displaying interaction using coplots in this response, Is interaction possible between two continuous variables?. – Andy W May 18 '12 at 17:58
1  
To Gung's answer I just want to say that statistical modeling involves noise which can disguise details in a polynomial regression model. i think that the centering issue that Bill Huber raised was a greta one because in one formualtion a linear term is missing and in the other it occurs with the quadratic term. The strength of the curvature in the signal dictates the need for a higher than first order term but really tells us nothing about the need for a linear term also. – Michael Chernick May 18 '12 at 23:58

@whuber's answer above is right on target in pointing out that omitting the linear term is the "usual" quadratic model is equivalent to saying, "I am absolutely certain that the extremum is at $x=0$."

However, you also need to check whether the software you are using has a "gotcha". Some software may automatically center the data when fitting a polynomial and testing its coefficients unless you turn off polynomial centering. That is, it may fit an equation that looks something like $Y = b_0 + b_2(x - \bar{x})^2$ where $\bar{x}$ is the mean of your $x$s. That would force the extremum to be at $x=\bar{x}$.

Your statement that both the linear and quadratic terms are significant when both are entered needs some clarification. For example, SAS may report a Type I and/or a Type III test for that example. Type I tests the linear before putting in the quadratic. Type III tests the linear with the quadratic in the model.

share|improve this answer
2  
This is a reasonable point, but just b/c the data were centered prior to creating $x^2$ doesn't mean you can be "absolutely certain that the extremum is at $x=0$". Saying that now is equivalent to having said "the extremum is at $x=\bar{x}$" before. In either case you are betting the unbiasedness of your model on your ability to specify the x-value of the extremum w/ infinite precision. The difference b/t Type I & Type III tests is also a potentially interesting addition, but nb, they would only differ if $x$ & $x^2$ are correlated, ie, if centering had not occurred. – gung May 22 '12 at 21:22
On a different note, you can refer to a user's contributions by stating their username, possible with the 'at' symbol. Eg, in this case, '@whuber's answer is right on target...' (A sentiment with which I agree.) – gung May 22 '12 at 21:24
1  
Thank you, Emil, for contributing those reminders: they are both worth bearing in mind. – whuber May 22 '12 at 23:05

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.