2
$\begingroup$

This is from Hollander's nonparametric inference, chapter 2:

  1. A multiple-choice quiz contains ten questions. For each question there are one correct answer and four incorrect answers. A student gets three correct answers on the quiz. Test the hypothesis that the student is guessing.

What is bugging me in this exercise is: how one would decide what the critical region should be? In R, of course, you could write:

 binom.test(3,10,0.2)

And you get a p-value of 0.4296. But what is the critical region behind this p-value and why is R deciding it?

I ran the following:

 pbinom(0:10,10,0.2)
 [1] 0.1073742 0.3758096 0.6777995 0.8791261 0.9672065 0.9936306
 [7] 0.9991356 0.9999221 0.9999958 0.9999999 1.0000000

 dbinom(3,10,0.2)
 [1] 0.2013266

So I thought a good critical region would be a two-sided one: if B=0 or B >= 4, reject the hypothesis that the student is guessing. The thing is, this doesn't seem to be a correct solution.

$\endgroup$
1
$\begingroup$
  1. Consider what's consistent with guessing and not guessing (and what we might really mean by "guessing" at all). It's a bit like assessing a claim of someone being psychic and able to predict the next die roll or something -- I would not be impressed if the psychic got an unusually low score since it's not consistent with the claim that they can predict.

    If a student isn't guessing, we should expect them to do better than guessing but not worse than it. So I would be looking only at rejecting if they get too many right, not if they get too few right.

  2. A critical region doesn't really "go with" a p-value; you choose your critical region before you see data, while the p-value is a function of the data. If you choose your critical region after the fact you're asking for people to accuse you of p-hacking.

    As you've discovered, it's easy to identify a p-value without having a critical region!

    With discrete test statistics and point nulls it makes sense to identify first what possible type I error rates there are.

    round(pbinom(0:9,10,.2,lower.tail=FALSE),4) # these will be offset by 1
    0.8926 0.6242 0.3222 0.1209 0.0328 0.0064 0.0009 0.0001 0.0000 0.0000 
    

    So as we see the reasonable significance levels seem to be ~12%, 3.3% or 0.64% with rejection regions of 4+, 5+ or 6+ respectively. Pick one!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.