2
$\begingroup$

I have two regressions:

(1)$$y=\beta_0+\beta_1x_1+u$$

(2) $$y=\beta_0+\beta_1x_1+\beta_2x_1^2+u$$

In regression (1) $\beta_1$ is a statistically significant determinant of $y$. In (2) we have the same regression except include a quadratic of $x_1$ which removes statistical significance from both the variables.

Why is this the case?

$\endgroup$
  • 1
    $\begingroup$ Could you try plotting $x_1$ against $y$ to look at their relationship? It could be that the relationship is not quadratic. $\endgroup$ – Hiromi Jun 16 '17 at 2:28
  • $\begingroup$ What method are you using to assess statistical significance? In particular, are you using anything like a Bonferroni correction? $\endgroup$ – Geoffrey Brent Jun 16 '17 at 2:29
  • $\begingroup$ @GeoffreyBrent Im just looking at p-values and t-tests. $\endgroup$ – EconJohn Jun 16 '17 at 2:35
  • $\begingroup$ You have to google for 'omitted variable bias'; leaving out a variable may introduce a bias in the coefficients of the other variables. In your case: dropping $x^2$ may lead to a biased estimate of the coefficient of $x$. $\endgroup$ – user83346 Jun 16 '17 at 3:39
1
$\begingroup$

If all the values of x are positive then there will be a correlation, often fairly strong, between $x$ and $x^2$. If so, then much of the variation might not be attributable to either one of them. As a result the F-test for the regression can be quite large but the individual tests of the two coefficients might indicate that they are both pretty worthless.

Make a graph of $x^2$ versus $x$ and compute their correlation.

You might overcome any correlation between them by centering $x$, that is, using z = x - x bar in place of x and $z^2$ in place of $x^2$.

In fitting polynomials it is standard practice to test each higher term conditional on the lower terms but not vice-versa. In your case this means that the linear term is significant, found when testing it alone, but the squared term is not, found when testing it with the linear term included.

$\endgroup$
  • $\begingroup$ "much of the variation might not be attributable to either one of them" - is that "not" meant to be there? $\endgroup$ – Geoffrey Brent Jun 17 '17 at 5:09
  • $\begingroup$ I did mean that it was not uniquely attributable to either one. Some part of the variation can't be attributed to either one unless either the linear and quadratic parts are orthogonal or the linear part, usually, is given credit first. $\endgroup$ – David Smith Jun 17 '17 at 12:53
  • $\begingroup$ ah right, that makes sense. I was thinking in terms of "the variation could equally well be attributable to either one". $\endgroup$ – Geoffrey Brent Jun 17 '17 at 14:22
  • $\begingroup$ English is a fine language, though its' informality can lead to confusion among honorable men and women. $\endgroup$ – David Smith Jun 17 '17 at 15:39
2
$\begingroup$

I can think of a couple of possibilities here. There are probably others.

One is that the software you're using to calculate p-values is applying something like a Bonferroni correction. This XKCD cartoon illustrates the problem nicely: the more statistics you look at, the greater the probability of finding a "significant" result purely by chance.

Bonferroni attempts to mitigate this by adjusting the threshold for significance according to the number of things you're checking for significance. In the first case, you have two parameters; in the second, you have three. So a value that appears "significant" when you have two parameters may no longer meet the tougher threshold that applies when you have three.

Another possibility is that your data can be described either as one weak linear effect, or as a combination of a weaker linear effect with an also-weak quadratic effect.

For example, say my data is clustered around (0,0) and (1,1). In a linear model, my best-fit approximation is y=x. In a quadratic model, I could end up with y=x, or with y=0.5x+0.5x^2, or various other possibilities; small differences in the inputs could make a large difference to the model parameters.

In this case, if you're replacing a weak linear effect with a weaker linear effect + a weak quadratic effect, you can end up dropping the "significance" of the linear effect a little bit below the cutoff.

If your data is such that x and x^2 are close to linearly dependent, you can expect poor behaviour generally.

Even small changes to a model can cause a p-value of 0.049 to change to 0.051 or vice versa. If we arbitrarily choose 0.05 as our cutoff, yes, that can cause a result to change from "significant" to "not significant".

$\endgroup$
  • 1
    $\begingroup$ Wait, lm() in R automatically does a Bonferroni correction? $\endgroup$ – EliK Jun 16 '17 at 14:45
  • $\begingroup$ I don't think the OP or I mentioned R specifically. I don't know what he's using so I have no idea whether it actually is applying Bonferroni, just suggesting it as a possibility. $\endgroup$ – Geoffrey Brent Jun 17 '17 at 5:08
1
$\begingroup$

When you include the squared term, the unsquared term represents the slope when x=0, which is probably not what you want. It is usually better to test sequentially. That is, don't include the squared term in the model when you are assessing the linear component.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.