I have a question regarding which test I should run on some data. I have small groups of people who answered questions. The answers to these questions have been classified as correct or incorrect. The N of the total number of questions asked is fairly large, however, the N of the total number of groups is quite small.

I'd like to analyse the accuracy based on group size. My initial thought was an ANOVA would be appropriate as I would divide the groups into size 1-2, 3-4, and 5+ and use overall group accuracy for each instance. However, as I said, the N is small. A colleague suggested that this is incorrect and a Chi Squared would be more appropriate where the N in question would be the instances of all the correct/incorrect responses instead of the overall group accuracy. Is either one of these or another test more appropriate?

  • $\begingroup$ Where possible you should avoid using the same symbol to mean more than one thing, and you should clearly define terms like 'accuracy' (is that just the proportion of correct answers?). $\endgroup$
    – Glen_b
    Jul 26, 2014 at 4:06

1 Answer 1


It seems you have used "N" in three different ways here so it is a bit confusing, but I will try to answer based on the following restatement: You have $n$ groups, and for the $i$th group, you have two measures $(s_i, y_i)$ representing, respectively, its size and its score (number of right answers) on a scale of $0$ to $N$. Moreover, each group was asked the same $N$ questions.

(Do I have that right?) If so, then $\ldots$

The first thing I'd do is make a scatterplot of the $y_i$ versus the $s_i$. Pictures really help understand what you have. If you don't see a pattern here, then there isn't much you can do with statistics, given that only group size $s_i$ is in play for predicting $y_i$.

While, technically, the $y_i$ are sums of Bernoulli ($0$ or $1$ values), the $\chi^2$ test is not really valid here unless the "success" probability is the same for each question -- which I seriously doubt. But if $N$ is large or even moderate, then the $y_i$ are just test scores, really, and it's pretty acceptable to just treat them as measurement data. You could do ANOVA, but it seems like a simple regression model with the $y_i$ as responses and the $s_i$ as predictor could do pretty well, possibly going to a quadratic or cubic fit if the straight line does not fit well.

If $n$ (my notation) is fairly small as you say, then you may not get anything definitive here. But I'm supposing that you meant it is small in terms of compiling a table of frequencies.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.