1
$\begingroup$

I have a test that has 39 questions. 20 of them have 18 possible answers and 19 of them have 3 possible answers. I built a QA system that answers 19 of the 20 and 18 of the 19 correctly. I want to know if this is significant from random. Normally I would do a binomial test for something like that.

I don't know how to incorporate the different correct likelihoods. I could do two binomial tests but I wouldn't want to increase my error chance. Is there a standard significance test for this kind of situation? Any help would be appreciated.

$\endgroup$

1 Answer 1

0
$\begingroup$

Each of the individual binomial tests will be far into the tail (both p-values will be way below one-in-a-million, indeed the first one will be incredibly tiny), so in this instance it will probably make little difference, but here goes:

I presume you want one-tailed tests.

You could combine the two sets of results in any of several ways. Here are some examples

  1. You could look at the total number correct. This is relatively easy to do the calculations for. You could do an exact convolution if you want it exactly (you only need the extreme upper tail so there's only a few values to worry about).

    If I did this right, the one-tailed p-value for this is about $4.65\times 10^{-30}$

  2. You could combine the two results into a test statistic by taking some linear combination*.

    * Since variance of each total under the null is pretty easy to calculate it should be possible to use "optimal" weights (weighted by precision -- the inverse of variance -- under the null). Or you could weight in any way that suited you.

  3. You could use Fisher's method for combining independent tests.

    If I did this right*, I think the p-value is about $6.3 \times 10^{-29}$

* this involves a chi-square approximation which won't be reasonable so far out into in the upper tail. One could compute an exact discrete p-value, but it will be similarly miniscule.

There are any number of other possibilities, depending on what you want to achieve or how you view the two sets of tests.

The second approach shouldn't be difficult to find a p-value for; it's just a bivariate discrete convolution to get the distribution of whatever linear combination you seek.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.