4
$\begingroup$

When creating nested models, does the addition of a predictor that turns out to be significant (Pr(>|t|) < 0.05) mean that the larger model is significantly better? If not, why not?

$\endgroup$

1 Answer 1

3
$\begingroup$

YES, BUT...

...it depends on what exactly you mean by "significant". It is the case that the p-value from t-testing that one added feature is the same as the p-value you would get from F-testing the nested models. In that sense, you might consider the larger model to be significantly better. In terms of practical significance, though, such a p-value tells you next to nothing, as p-values can be driven small despite tiny effect sizes when the sample size is large.

Let's look at a simulation to show that the t-based and F-based p-values are the same.

set.seed(2023)
N <- 100
x1 <- runif(N)
x2 <- x1 + runif(N)
y <- x1 - 0.3*x2 + rnorm(N)
L0 <- lm(y ~ x1)
L1 <- lm(y ~ x1 + x2)
summary(L1)$coef # t-based p-value is 0.0959419435
anova(L0, L1)    # F-based p-value is 0.09594

Aside from rounding, the p-values are equal.

$\endgroup$
2
  • $\begingroup$ Thank you! In consequence, does it mean that the sample size calculation for a nested model comparison is the same as for the larger model alone? $\endgroup$
    – drat
    Commented Jan 30, 2023 at 20:19
  • 1
    $\begingroup$ The sample size is the same, yes. You just consider more features of the observations in your sample. @drat $\endgroup$
    – Dave
    Commented Jan 30, 2023 at 20:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.