Constructing the critical region of a binomial test

Question

This is from Hollander's nonparametric inference, chapter 2:

7. A multiple-choice quiz contains ten questions. For each question there are one correct answer and four incorrect answers. A student gets three correct answers on the quiz. Test the hypothesis that the student is guessing.

What is bugging me in this exercise is: how one would decide what the critical region should be? In R, of course, you could write:

 binom.test(3,10,0.2)

And you get a p-value of 0.4296. But what is the critical region behind this p-value and why is R deciding it?

I ran the following:

 pbinom(0:10,10,0.2)
 [1] 0.1073742 0.3758096 0.6777995 0.8791261 0.9672065 0.9936306
 [7] 0.9991356 0.9999221 0.9999958 0.9999999 1.0000000

 dbinom(3,10,0.2)
 [1] 0.2013266

So I thought a good critical region would be a two-sided one: if B=0 or B >= 4, reject the hypothesis that the student is guessing. The thing is, this doesn't seem to be a correct solution.

Answer 1

1. Consider what's consistent with guessing and not guessing (and what we might really mean by "guessing" at all). It's a bit like assessing a claim of someone being psychic and able to predict the next die roll or something -- I would not be impressed if the psychic got an unusually low score since it's not consistent with the claim that they can predict.

   If a student isn't guessing, we should expect them to do better than guessing but not worse than it. So I would be looking only at rejecting if they get too many right, not if they get too few right.

2. A critical region doesn't really "go with" a p-value; you choose your critical region before you see data, while the p-value is a function of the data. If you choose your critical region after the fact you're asking for people to accuse you of p-hacking.

   As you've discovered, it's easy to identify a p-value without having a critical region!

   With discrete test statistics and point nulls it makes sense to identify first what possible type I error rates there are.

   round(pbinom(0:9,10,.2,lower.tail=FALSE),4) # these will be offset by 1
   0.8926 0.6242 0.3222 0.1209 0.0328 0.0064 0.0009 0.0001 0.0000 0.0000 
   

   So as we see the reasonable significance levels seem to be ~12%, 3.3% or 0.64% with rejection regions of 4+, 5+ or 6+ respectively. Pick one!