# How to calculate type II error between glm fits in R?

I'm using a GLM with logistic link function to try to predict Y (0 or 1) as a function of a ton of predictor variables (A, B, C, etc.). Some of the predictor variables (A*, B*, C*, etc.) have been shown in other studies to be significant predictors. I want to show essentially that Y is unrelated to all of the other predictor variables, and I thought the simplest way to do this would be to run the full model (Y ~ .) and the null model (Y ~ A* + B* + C* + ...), and then use anova() to compare the two and show that they aren't different (i.e. have equal predictive power).

However, anova() only outputs p-values (type I error), but I need a type II error rate here (since I want to show that the models are the same, I need a false negative rate for that). Any ideas on how to approach this?

• See if the Anova function in the car package does what you want. Dec 11, 2013 at 16:32
• Part of the reason I couldn't find much help online is because there is a "type II" way that the built-in anova() function (and Anova() in the car package) runs. Unfortunately, that "type II" is completely different from "type II error", which is what I need.
– Zack
Dec 11, 2013 at 17:05
• Are you after a type II error rate? Dec 11, 2013 at 17:30
– Zack
Dec 11, 2013 at 19:08

However, anova() only outputs p-values (type I error), but I need a type II error rate here (since I want to show that the models are the same, I need a false negative rate for that).

No I don't think you do.

The anova is tests the hypothesis that the more complex model fits the observed data no better than the (nested) simpler model. If you fail to reject this hypothesis then you've gone some way to showing that it is reasonable to conclude that the models are actually 'the same'.

The type 2 error rate refers to how often this anova would assert that the models were 'the same' when they were in fact different. This interesting, because it tells you to something about the magnitude of differences the test could reliably pick up if there were any, but it doesn't so directly answer your question.

More generally you might find it helpful to distinguish between Fisher's $p$, a (post experiment) measure not equivalent to Neyman and Pearson's $\alpha$ which is the (pre experimental) Type 1 error rate. Base R's anova function is, unsurprisingly, set up for Fisherian interpretation.

MacCallum et al. Testing Differences Between Nested Covariance Structure Models: Power Analysis and Null Hypotheses present a method for calculating power (i.e. type II error rate) for likelihood ratio testing of two models.

• Hi Matt, would you care to a more self contained description of how the method works?
– Momo
Mar 16, 2014 at 9:48