What is the correct test to use for binary classification when the subject knows the proportion?

Question

I have a sample of $N=30$ questions that the test subject is going to rate as either True (T) or False (F). The subject is informed that the ratio of T/F answers is exactly 50%. We can assume that the subject will adjust his answers so that he/she will meet the 50% ratio (and thus would never score 17 T and 13 F).

Question: What would be the correct test to reject the null hypothesis that the subject can't distinguish between the two classes with a $p=0.05$?

Thoughts: There are $30!/(15! 15!) \approx 1.5*10^8$ different permutations of the 15(T) and 15(F). One could work out the number of ways to get exactly $k=0,2,4,...,30$ correct and from there construct a test. I have a feeling this problem has already been solved before and has a name... I am interested for the specific case and the general case when $N$ is arbitrary and the ratio differs from 1/2.

Answer 1

This is called a binomial test. You've got the idea of it already, but it does in fact have a name.

Binomial Test

You essentially calculate the probability that the event that occurred (or something more extreme) would happen given the data are generated from the known binomial distribution and that's your p-value.

As Hooked stated in his comment, the standard two-sided interpretation will reject if the observed value is high or low. Make sure you're interpreting the test correctly if you're just trying to do a one sided test.

Answer 2

I don't think the fact that the proportion of "True" guesses is constrained to be 50% changes the statistical problem in any meaningful/complicated way. I think the null hypothesis that subjects can't distinguish between the two classes can still be tested using a pretty standard logistic regression model.

Specifically, let the outcome be a vector of 0s and 1s that indicate whether a subject responded with "False" or "True," and regress that on a predictor consisting of -1/2 if "False" is the correct answer and +1/2 if "True" is the correct answer. The slope from this logistic regression gives the increase in log-odds of responding correctly vs. responding incorrectly. Chance performance implies that this coefficient should be 0. And the constraint that you mentioned in your question simply makes it so that we know in advance that the intercept will be 0. So we could technically omit the intercept if we wanted, but I think it shouldn't matter much.

If you do this as a probit regression rather than a logistic regression, then it's equivalent to fitting the standard, equal-variance, two-choice signal detection theory model. So fitting this model with random effects for subjects would amount to fitting a multilevel signal detection model. See the following reference:

• Wright, D. B., Horry, R., & Skagerberg, E. M. (2009). Functions for traditional and multilevel approaches to signal detection theory. Behavior Research Methods, 41(2), 257-267. PDF

Answer 3

The key realization in this problem is that answering each question can be considered to be a Bernoulli problem with probability $p=0.5$.

We want to see at what point we start to believe that the student is not simply flipping a coin, and has actually done some studying. We know that he or she could potentially get all the questions right just by luck (of course not in this world, but still...), so we'll need to set a cutoff level above which we are ready to accept a certain risk of being duped - let's go with the traditional $5\%$, which will be our risk $\alpha$.

We want, then, to find out the number of "successes" ($x$) in answering the questions that would only be surpassed randomly in $5\%$ of the cases. And here there is a very pesky bend in the path: notice that "success" is getting the answer right, which is different from the answer being True. It is very fortunate that this doesn't matter because of the $50/50$ split.

We want to calculate what number of successes out of $30$ trials will amount to a $95\%$ cumulative probability:

$.95 = \displaystyle \sum_{1}^{x} {30 \choose x}\,0.5^x\,0.5^{(30-x)}=\displaystyle \sum_{1}^{x} {30 \choose x}\,0.5^{30}$. And solve for $x$,

Using [R] this can be done running the function qbinom(0.95, 30, prob = 0.5, lower.tail = T) = 19, which implies that the probability of getting by chance more than $20$ questions right is virtually $0.5$, as can be shown with the command pbinom(19, 30, prob = 0.5, lower.tail = F) = 0.04936.

So if we want to "make sure" that there's been studying done, we want to set the cutoff limit at the nice and round figure of $20$ questions right.