0
$\begingroup$

I'm using a GLM with logistic link function to try to predict Y (0 or 1) as a function of a ton of predictor variables (A, B, C, etc.). Some of the predictor variables (A*, B*, C*, etc.) have been shown in other studies to be significant predictors. I want to show essentially that Y is unrelated to all of the other predictor variables, and I thought the simplest way to do this would be to run the full model (Y ~ .) and the null model (Y ~ A* + B* + C* + ...), and then use anova() to compare the two and show that they aren't different (i.e. have equal predictive power).

However, anova() only outputs p-values (type I error), but I need a type II error rate here (since I want to show that the models are the same, I need a false negative rate for that). Any ideas on how to approach this?

Improve this question
$\endgroup$
4
  • $\begingroup$ See if the Anova function in the car package does what you want. $\endgroup$
    – Thomas
    Dec 11, 2013 at 16:32
  • $\begingroup$ Part of the reason I couldn't find much help online is because there is a "type II" way that the built-in anova() function (and Anova() in the car package) runs. Unfortunately, that "type II" is completely different from "type II error", which is what I need. $\endgroup$
    – Zack
    Dec 11, 2013 at 17:05
  • 1
    $\begingroup$ Are you after a type II error rate? $\endgroup$
    – Michael M
    Dec 11, 2013 at 17:30
  • $\begingroup$ Yes, sorry about that! $\endgroup$
    – Zack
    Dec 11, 2013 at 19:08

2 Answers 2

Reset to default
2
$\begingroup$

However, anova() only outputs p-values (type I error), but I need a type II error rate here (since I want to show that the models are the same, I need a false negative rate for that).

No I don't think you do.

The anova is tests the hypothesis that the more complex model fits the observed data no better than the (nested) simpler model. If you fail to reject this hypothesis then you've gone some way to showing that it is reasonable to conclude that the models are actually 'the same'.

The type 2 error rate refers to how often this anova would assert that the models were 'the same' when they were in fact different. This interesting, because it tells you to something about the magnitude of differences the test could reliably pick up if there were any, but it doesn't so directly answer your question.

More generally you might find it helpful to distinguish between Fisher's $p$, a (post experiment) measure not equivalent to Neyman and Pearson's $\alpha$ which is the (pre experimental) Type 1 error rate. Base R's anova function is, unsurprisingly, set up for Fisherian interpretation.

Improve this answer
$\endgroup$
0
$\begingroup$

MacCallum et al. Testing Differences Between Nested Covariance Structure Models: Power Analysis and Null Hypotheses present a method for calculating power (i.e. type II error rate) for likelihood ratio testing of two models.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Hi Matt, would you care to a more self contained description of how the method works? $\endgroup$
    – Momo
    Mar 16, 2014 at 9:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.