# What is the correct test to use for binary classification when the subject knows the proportion?

I have a sample of $N=30$ questions that the test subject is going to rate as either True (T) or False (F). The subject is informed that the ratio of T/F answers is exactly 50%. We can assume that the subject will adjust his answers so that he/she will meet the 50% ratio (and thus would never score 17 T and 13 F).

Question: What would be the correct test to reject the null hypothesis that the subject can't distinguish between the two classes with a $p=0.05$?

Thoughts: There are $30!/(15! 15!) \approx 1.5*10^8$ different permutations of the 15(T) and 15(F). One could work out the number of ways to get exactly $k=0,2,4,...,30$ correct and from there construct a test. I have a feeling this problem has already been solved before and has a name... I am interested for the specific case and the general case when $N$ is arbitrary and the ratio differs from 1/2.

• If you fix the margins (you know the number of T and F) this is hypergeometric -- exactly like the problem of the lady tasting tea, in fact. She has 8 cups, where four are milk-first and tries to identify which are which (pick the four milk-first cups). This is equivalent to identifying four T answers out of 8, a special case of the problem I think you're describing. The most common test is the usual chi-square, but there's also the G-test and Fisher exact test (which uses the hypergeometric). – Glen_b Nov 4 '15 at 1:14
• @Glen_b Dang, I came here to say that! – Sycorax Nov 4 '15 at 1:15
• @user777 I watched this a while waiting for someone to say something, but decided I had better speak up. Sorry. – Glen_b Nov 4 '15 at 1:17
• @Glen_b thanks for the reference to the problem and the corresponding history learned from the wikipedia page. – Hooked Nov 4 '15 at 1:36
• To my disgust the term "hypergeometric distribution" - the name of the distribution of the number of successes in this problem - doesn't occur anywhere on that Wikipedia page on the lady tasting tea (it has a link to the binomial! Seriously, what?!). The Wikipedia stats pages reach a new low. You might as well read Yahoo answers. (Edit: Well, I fixed that issue on the wikipedia page but it feels like sticking a finger in a dyke while the sea crashes over the top.) – Glen_b Nov 4 '15 at 1:56

This is called a binomial test. You've got the idea of it already, but it does in fact have a name.

Binomial Test

You essentially calculate the probability that the event that occurred (or something more extreme) would happen given the data are generated from the known binomial distribution and that's your p-value.

As Hooked stated in his comment, the standard two-sided interpretation will reject if the observed value is high or low. Make sure you're interpreting the test correctly if you're just trying to do a one sided test.

• Obviously that's what it's called. It's been a long day if I didn't think to search for that. Thanks! – Hooked Nov 3 '15 at 22:21
• I just looked it up and you may want to add it to your answer, but the two sided p-value test for p<0.05 is getting either 21 or 9 correct. – Hooked Nov 3 '15 at 22:25

I don't think the fact that the proportion of "True" guesses is constrained to be 50% changes the statistical problem in any meaningful/complicated way. I think the null hypothesis that subjects can't distinguish between the two classes can still be tested using a pretty standard logistic regression model.

Specifically, let the outcome be a vector of 0s and 1s that indicate whether a subject responded with "False" or "True," and regress that on a predictor consisting of -1/2 if "False" is the correct answer and +1/2 if "True" is the correct answer. The slope from this logistic regression gives the increase in log-odds of responding correctly vs. responding incorrectly. Chance performance implies that this coefficient should be 0. And the constraint that you mentioned in your question simply makes it so that we know in advance that the intercept will be 0. So we could technically omit the intercept if we wanted, but I think it shouldn't matter much.

If you do this as a probit regression rather than a logistic regression, then it's equivalent to fitting the standard, equal-variance, two-choice signal detection theory model. So fitting this model with random effects for subjects would amount to fitting a multilevel signal detection model. See the following reference:

• Wright, D. B., Horry, R., & Skagerberg, E. M. (2009). Functions for traditional and multilevel approaches to signal detection theory. Behavior Research Methods, 41(2), 257-267. PDF

The key realization in this problem is that answering each question can be considered to be a Bernoulli problem with probability $p=0.5$.

We want to see at what point we start to believe that the student is not simply flipping a coin, and has actually done some studying. We know that he or she could potentially get all the questions right just by luck (of course not in this world, but still...), so we'll need to set a cutoff level above which we are ready to accept a certain risk of being duped - let's go with the traditional $5\%$, which will be our risk $\alpha$.

We want, then, to find out the number of "successes" ($x$) in answering the questions that would only be surpassed randomly in $5\%$ of the cases. And here there is a very pesky bend in the path: notice that "success" is getting the answer right, which is different from the answer being True. It is very fortunate that this doesn't matter because of the $50/50$ split.

We want to calculate what number of successes out of $30$ trials will amount to a $95\%$ cumulative probability:

$.95 = \displaystyle \sum_{1}^{x} {30 \choose x}\,0.5^x\,0.5^{(30-x)}=\displaystyle \sum_{1}^{x} {30 \choose x}\,0.5^{30}$. And solve for $x$,

Using [R] this can be done running the function qbinom(0.95, 30, prob = 0.5, lower.tail = T) = 19, which implies that the probability of getting by chance more than $20$ questions right is virtually $0.5$, as can be shown with the command pbinom(19, 30, prob = 0.5, lower.tail = F) = 0.04936.

So if we want to "make sure" that there's been studying done, we want to set the cutoff limit at the nice and round figure of $20$ questions right.

• @Hooked Take a look, I think the result is different from some of the comments. Let me know if it helps. – Antoni Parellada Nov 4 '15 at 0:50
• Thanks for the answer! I appreciate the detailed explanation but to be clear, this is the same test as the one referred to in @MentatOfDune 's answer correct? – Hooked Nov 4 '15 at 1:32
• @Hooked We are working with the binomial distribution, yes. I simply was trying to convey a bit the nuance of what value is cut out and which is included (19, 20, 21?). This is an important point in a discrete distribution, and as it relates to [R] it changes from qbinom to binom.test, for instance. Also I couldn't quite get the two sided test idea. But, hey, it sounds as though you truly have a very good handle on the overall issue with calculating combinations, etc. right from your OP, so I hope I didn't get in your way with all the [R]-specific stuff, and possibly, that I helped some. – Antoni Parellada Nov 4 '15 at 1:39