p-value is missing from drop1 (R)

Question

I have the following model model (R language). My goal model is Y~N*V but since this also happens for Y~N+V I thought I might change to the simpler model.

model <- lm(Y~N+V,data=data)

and running drop1(model,REML=FALSE) I get the output:

Model:
Y ~ N + V
       Df Sum of Sq       RSS    AIC
<none>               88592297 1027.7
N       5  30480453 119072749 1038.9
V       3  89885035 178477332 1072.1

The p-values are missing. I don't think it's that the model is oversaturated because the dataset has 72 observations and factor N has 6 levels, and factor V has 4 levels. So 1+4+6 < 72 observations in the dataset.

EDIT I manually selected the test to be used via drop1(model,test="F") and now it gives the p-values. But is this the correct test?

Answer 1

With a linear model lm you have four options resulting in different hypothesis testing

• test="none" in this case no hypothesis test is performed.

  Why would someone use drop1 without hypothesis testing, what's the point? The answer is that drop1 is not only useful for hypothesis testing, and it can also be used for model selection. With model selection one uses different measures than p-values. A hypothesis test often favours the null hypothesis and is conservative towards including variables. With a measure like AIC you can include variables when the p-value is around 0.157, which is more liberal than the 0.05 or smaller cut-off values in most hypothesis tests (see Why can't we use AIC and p-value variable selection within the same model building exercise?).

• test="F" in this case an F-test is performed.

  This sort of testing is much analogous to the F-test used for ANOVA but now for a linear model. A comparison is made for two variances 1. the variance in the estimated values and 2. the variance between the estimated values and the observed values (the residuals).

  This test could be very well appropriate for your situation. One thing to consider are the assumptions. These are equality of variance and the random part that is normal distributed. (These assumptions will not be different for the other options, if the assumptions are not within bounds then you should use something else than lm with drop1)

• test = Chisq in this case a chi-squared test is performed. This has two variations

  • With a defined scale. If you know the standard deviation of the error distribution (for instance based on theoretical grounds or from earlier experiments) then this test will be like a F-test but without the comparison of two variances. The statistic in the F-test compares a ratio of residual sum of squares $$F = \frac{\left(\frac{RSS_1 - RSS_2}{p_2-p_1}\right)}{\left(\frac{RSS_2}{n-p_2}\right)}$$ The $\chi^2$ is similar but with a defined scale it replaces the denominator with a fixed expression and the scaling by the parameters is left out $$\chi^2 = \frac{{RSS_1 - RSS_2}}{\sigma^2}$$ This expression will be $\chi^2$ distributed if the null hypothesis is true (ie. if the true model coefficient equals zero).

    This test is probably not appropriate for you. Or at least, you did not mention whether the standard deviation is known. If this is indeed not known then you can not perform this test.

  • Without defined scale. If you do not specify the scale, then a likelihood ratio test will be performed. This is a more general test that works with many different types of models. In your case for a linear model it assumes a Gaussian distribution for the error terms and effectively you are computing some sort of difference in squared residuals. If you work out the likelihood ratio then you end up with $$\chi^2 = n\log \left(1 + \frac{(RSS_1-RSS_2)}{RSS_2} \right) \approx \frac{(RSS_1-RSS_2)}{RSS_2/n} $$ where the right side is the first term from a Taylor series expansion. So when $n$ is large this expression will approach a chi squared distribution for large $n$. Or in fact, it looks more like the F-statistic, but that will actually also approach a chi squared distribution for large $n$.

    This test is not appropriate for you. It is an approximation. The statistic is only following a chi squared distribution in the limit $n \to \infty$. Since the F-test is more precise there is no reason that you need to use this likelihood ratio test.

Answer 2

For linear models, as returned by the R function lm, the test based on the F-distribution is allright, because it uses a test statistic that is F-distributed. Note that for continuous variables, the p-values obtained by drop1 with the F-test (aka "Type II(I) ANOVA") are identical to the p-values based on the t-statistic returned by summary(lm).

For generalized linear models, as returned by the R function glm, you must use test="Chisq". The resulting p-values are based on log-likelihood differences, which are asymptotically $\chi^2$-distributed, according to an old result by Wilk from 1938. In principle, you could also use this for linear models, but as it is only an asymptotic approximation, there is no benefit in using it instead of the F-statistic based on $R^2$ differences.