4
$\begingroup$

When regression coefficient is nearly 0 (in fact in the real model it's exactly 0), what's the meaning of p value (<0.05) of the coefficient?

For example, I did a multiple variable regression with simulated data in R with lm().

Generate simulation data with the equation $$ y=2x_1^2+3x_2^2+3x_1+5 $$

The terms $x_1x_2$ and $x_2$ coefficients are zero. Using the data to do regression.

xmesh=mesh(seq(-4,4,0.1),seq(-4,4,0.1))
x1=as.vector(xmesh$x)
x2=as.vector(xmesh$y)
y=2*x1^2+3*x2^2+3*x1+5
model=lm(y~x1+x2+I(x1^2)+I(x2^2)+I(x1*x2)) 
summary(model)

The result is :

Call:
lm(formula = y ~ x1 + x2 + I(x1^2) + I(x2^2) + I(x1 * x2))

Residuals:
       Min         1Q     Median         3Q        Max 
-8.871e-12 -4.500e-15 -7.000e-16  5.700e-15  4.194e-12 

Coefficients:
              Estimate Std. Error    t value Pr(>|t|)    
(Intercept)  5.000e+00  3.301e-15  1.515e+15  < 2e-16 ***
x1           3.000e+00  7.545e-16  3.976e+15  < 2e-16 ***
x2          -3.348e-15  7.545e-16 -4.438e+00 9.22e-06 ***
I(x1^2)      2.000e+00  3.609e-16  5.542e+15  < 2e-16 ***
I(x2^2)      3.000e+00  3.609e-16  8.314e+15  < 2e-16 ***
I(x1 * x2)  -9.377e-16  3.227e-16 -2.906e+00  0.00367 ** 
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 1.429e-13 on 6555 degrees of freedom
Multiple R-squared:     1,  Adjusted R-squared:     1 
F-statistic: 2.313e+31 on 5 and 6555 DF,  p-value: < 2.2e-16

We can see that the coefficients of term $x_2$ and $x_1x_2$ are nearly 0, and p-value<0.01. I think that lm() did the significance test of coefficient based on t-test with NULL hypothesis $\beta=0$. So p-value<0.05 should mean that the coefficient is significantly different to 0. However the coefficient should be 0 in my model. I am confused. How to interpret these two coefficients' significance?

Add a new test $y=2x_1^2+3x_1+0.001x_2+5$

> y2=2*x1^2+3*(x1)+5+0.001*x2
> model3=lm(y2~x1+x2+I(x1^2)+I(x2^2)+I(x1*x2)) 
> summary(model3)

Call:
lm(formula = y2 ~ x1 + x2 + I(x1^2) + I(x2^2) + I(x1 * x2))

Residuals:
       Min         1Q     Median         3Q        Max 
-9.237e-12 -1.700e-15 -1.000e-16  2.200e-15  2.757e-12 

Coefficients:
              Estimate Std. Error    t value Pr(>|t|)    
(Intercept)  5.000e+00  2.840e-15  1.761e+15   <2e-16 ***
x1           3.000e+00  6.492e-16  4.621e+15   <2e-16 ***
x2           1.000e-03  6.492e-16  1.540e+12   <2e-16 ***
I(x1^2)      2.000e+00  3.105e-16  6.441e+15   <2e-16 ***
I(x2^2)     -2.722e-16  3.105e-16 -8.770e-01    0.381    
I(x1 * x2)  -3.226e-16  2.776e-16 -1.162e+00    0.245    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 1.229e-13 on 6555 degrees of freedom
Multiple R-squared:     1,  Adjusted R-squared:     1 
F-statistic: 1.257e+31 on 5 and 6555 DF,  p-value: < 2.2e-16

You can see that in the new test, the coefficients and std Error of term $x_1x_2$ and $x_2^2$ are essentially zero. Their p-value are large enough to accept the null hypothesis that $\beta=0$, it's a good result.

How to interpret the p-value of essentially zero coefficients in the two tests?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
10
$\begingroup$

This has more to do with how computers work than with p-values. You have to remember that computers can't represent real numbers exactly. We are dealing with floating point numbers. So some algorithms will never get exactly zero, even if analytically the result should be zero. For example (0.3-0.2) - (0.2-0.1) will not give you zero.

You can see that the estimates are essentially zero: 

all.equal(-3.348e-15, 0)
TRUE
all.equal(-9.377e-16, 0)
TRUE

The same goes for your standard errors: they are zero.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for your answer. I did neglect the limit of computer floating point. So I think that due to the reported coefficient and standard errors are essentially zero, the reported p-value is meaningless. Is that right? $\endgroup$
    – Frank Yan
    Commented May 5, 2014 at 7:36
  • $\begingroup$ I added a new test in question post. In the new test, although two terms essentially equal 0, but their p-values are quite different from test1. In test2, their p value can be used to accept the null hypothsis, so we can drop the two terms. I think they are meanigful. Maybe as @usεr11852 said, the finite sample effects is the reason for p-value in test1 which reject the null hypothesis. What do you think about it? $\endgroup$
    – Frank Yan
    Commented May 5, 2014 at 8:18
  • $\begingroup$ Your model is deterministic so what is driving these results is the floating point precision error.I would guess that floating point errors are not well behaved (it seems like this inspecting the residuals of the model). So assuming we did not know that your model was deterministic and we wanted to take this floating point error as a source of uncertainty, the normal approximation would not be good - so the p-values would be wrong anyway. You would have to model the error properly, maybe with fat tails for example. $\endgroup$
    – Carlos Cinelli
    Commented May 5, 2014 at 12:09
  • $\begingroup$ But you shouldn't take p-values as your sole input for decision. In most problems a coefficient of -3.348e-15 is meaningless, regardless of the the p-value! If you needed that kind of precision, then you would have to use the proper instrument. $\endgroup$
    – Carlos Cinelli
    Commented May 5, 2014 at 12:11
1
$\begingroup$

Two points to be made:

  1. Floating point arithmetic. You are not working with arbitrary precision arithmetic in your system so values of that magnitude are treated as essentially zero. You are almost surely using some implementation of BLAS \ LAPACK on working with doubles to solve the QR-decomposition that corresponds to your regression problem.
  2. Finite sample effects. Given that your interaction terms are not "entirely" unrelated minuscule regularities might be detected. That's why the $p$-values might not be exactly zero. It pretty much the same effect as adding redundant covariates in an OLS problem and seeing $R^2$ increasing.

The $\beta$ coefficients you examine can be safely interpreted as being zero.

(Ah, as I was writing this @carloscinelli posted the same essentially; good man, +1. :) )

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for your answer. I added a new test in the question post which I reduced the effects of x2 term. I think maybe the finite sample effects are the main cause of rejecting null hypothesis. In test 2, as I reduced the related effects, it indeed accepted the null hypothesis. So what's your opinion? $\endgroup$
    – Frank Yan
    Commented May 5, 2014 at 8:29
  • $\begingroup$ Yes you are correct. Reducing the influence of $x2$ is what helped you reject the null. Let me stress in the original test you still had the same inferential result just differently presented. The coefficients for $x2$ and $x1*x2$ were still zero for all practical purposes, ie. believing that "truly" the $\beta$ coefficient of a parameter to be -9.377e-16 does little more than saying it is zero; $H_0$ and $H_a$ are the same. $\endgroup$
    – usεr11852
    Commented May 5, 2014 at 12:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.