1
$\begingroup$

I am trying to compare two models:

Costs = Income + Education1 + ... + Education10 and Costs = Income

From the ANOVA table using anova() in conjunction with the corresponding lm() objects, I get that the p-value is < 2.2e-16. However, I was wondering if there was any way that the numerous factor levels in Education could throw off the p-value? I was thinking that it may be deflated due to multiple comparisons because of the numerous parameters but I wasn't sure. Also, is this test influenced by multicollinearity between Income and Education?

Thanks

$\endgroup$
4
  • 3
    $\begingroup$ Which p-value is <2.2e-16? // What multiple comparisons do you see being made? I see one comparison being made: the full model (with education) compared to the restricted model (only income). $\endgroup$
    – Dave
    Commented Dec 2, 2022 at 5:37
  • $\begingroup$ One effect that might make the F-test a false result could be a low degree of freedom for the residuals in combination with a non-normal distribution of the error terms. In that case the approximation of the F-statistic by a F-distribution might be less accurate. $\endgroup$ Commented Dec 2, 2022 at 8:22
  • $\begingroup$ Could you post the entire results/table from the ANOVA. $\endgroup$ Commented Dec 2, 2022 at 8:22
  • $\begingroup$ Re the wording "Can an F-test always be trusted?" - the test in itself is mathematically well defined and, as long as you compute it correctly, always does what it's defined to do. This is not an issue of "trust". What may be trusted or not is the interpretation that you give the result, which you haven't explained. Other than that, in your current description there is nothing that invalidates the F-test, although there may be other reasons for concern such as potential violations of the model assumptions that we can't see from your question. $\endgroup$ Commented Dec 2, 2022 at 10:23

1 Answer 1

3
$\begingroup$

As to the first question, I agree with Dave that the F test does not perform multiple comparisons, but just one. You would possibly run into a multiple comparison issue if you did separate t tests on each education level to find out which has a nonzero effect.

With the F test, you "only" have the alternative hypothesis that there is at least one level that has a nonzero effect, but if you reject, the "price" you pay for using the F test is that you do not know which level is "responsible" for the rejection.

As to the second question, somewhat to my surprise (but see the comments below), this little simulation does not find any sensitivity of the F test to what is near perfect multicollinearity when we test for the relevance of all correlated predictors (see below for an edit).

library(lmtest)
library(mvtnorm)

n <- 30
reps <- 5000
k <- 2
critical.value <- qf(p = .95, df1 = k, df2 = n-k)

covariance <- 0.999
Fstat <- rep(NA,reps)

for (i in 1:reps){
  y <- rnorm(n)
  X <- rmvnorm(n, sigma = matrix(c(1,covariance,covariance,1), ncol=2))
  reg <- lm(y~X)
  Fstat[i] <- waldtest(reg, test="F")$F[2] 
}

mean(Fstat>critical.value)

hist(Fstat, breaks = 100, col="lightblue", freq = F, xlim=c(0,5))
x <- seq(0, 6, by=.01)
lines(x, df(x, df1 = k, df2 = n-k), lwd=2, col="purple")

The rejection rate (mean(Fstat>critical.value)) is, even for relatively small $n=30$ and very strong covariance of the regressors (0.999), still very close to the nominal 0.05, and the histogram and theoretical null density agree very well:

enter image description here

EDIT: Here is a modification of the code that, in response to Dave's comment , tests only one of the two correlated predictors (so that the F statistic is then nothing but a squared t-statistic), and the result still appears to go throgh:

library(lmtest)
library(mvtnorm)

n <- 30
reps <- 5000
k <- 2
critical.value <- qf(p = .95, df1 = k-1, df2 = n-k) 


covariance <- 0.999
Fstat <- rep(NA,reps)

for (i in 1:reps){
  y <- rnorm(n)
  X <- rmvnorm(n, sigma = matrix(c(1,covariance,covariance,1), ncol=2))
  X1 <- X[,1]
  X2 <- X[,2]
  reg <- lm(y~X1+X2)
  
  regX1 <- lm(y~X1)
  Fstat[i] <- waldtest(reg, regX1, test="F")$F[2] 
}

mean(Fstat>critical.value)

hist(Fstat, breaks = 200, col="lightblue", freq = F, xlim=c(0,5))
x <- seq(0, 6, by=.01)
lines(x, df(x, df1 = k-1, df2 = n-k), lwd=2, col="purple")
$\endgroup$
5
  • $\begingroup$ Isn’t that simulation testing the correlated features together rather than having one correlated feature in the restricted model (and both in the full model)? $\endgroup$
    – Dave
    Commented Dec 2, 2022 at 8:19
  • $\begingroup$ Yes, it does indeed. I would have still expected that the estimate of the full model will then be so noisy that we get less precise behavior of the F test in finite samples. I will play around with the MC and report back! $\endgroup$ Commented Dec 2, 2022 at 8:29
  • $\begingroup$ @Dave, please see my edit - still no effect of strong multicollinearity... $\endgroup$ Commented Dec 2, 2022 at 8:49
  • 1
    $\begingroup$ The theory behind the F-test doesn't rely on low correlations in any way, so it should be no surprise that it works in a situation in which its model assumptions are perfectly fulfilled. Of course estimators are more noisy, but this is taken into account by the test statistic and its distribution theory. (Power will be somewhat lower with the same regression coefficients compared to a zero correlation situation, but this shouldn't surprise anyone either.) $\endgroup$ Commented Dec 2, 2022 at 10:03
  • $\begingroup$ Sure, in principle that is clear, but I found it surprising that the theory works so well in finite sample near a "boundary" of a parameter space. I had persistent AR processes in my mind, where local to unity theory often works much better than that for stationary processes. To be sure, here, we have exact distribution theory, where the unit root example relies on asymptotic theory. $\endgroup$ Commented Dec 2, 2022 at 10:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.