Appropriate Statistical Test for multiple groups I have a question regarding which test I should run on some data.  I have small groups of people who answered questions.  The answers to these questions have been classified as correct or incorrect.  The N of the total number of questions asked is fairly large, however, the N of the total number of groups is quite small.
I'd like to analyse the accuracy based on group size.  My initial thought was an ANOVA would be appropriate as I would divide the groups into size 1-2, 3-4, and 5+ and use overall group accuracy for each instance.  However, as I said, the N is small.  A colleague suggested that this is incorrect and a Chi Squared would be more appropriate where the N in question would be the instances of all the correct/incorrect responses instead of the overall group accuracy.  Is either one of these or another test more appropriate?
 A: It seems you have used "N" in three different ways here so it is a bit confusing, but I will try to answer based on the following restatement: You have $n$ groups, and for the $i$th group, you have two measures $(s_i, y_i)$ representing, respectively, its size and its score (number of right answers) on a scale of $0$ to $N$. Moreover, each group was asked the same $N$ questions.
(Do I have that right?) If so, then $\ldots$
The first thing I'd do is make a scatterplot of the $y_i$ versus the $s_i$. Pictures really help understand what you have. If you don't see a pattern here, then there isn't much you can do with statistics, given that only group size $s_i$ is in play for predicting $y_i$.
While, technically, the $y_i$ are sums of Bernoulli ($0$ or $1$ values), the $\chi^2$ test is not really valid here unless the "success" probability is the same for each question -- which I seriously doubt. But if $N$ is large or even moderate, then the $y_i$ are just test scores, really, and it's pretty acceptable to just treat them as measurement data. You could do ANOVA, but it seems like a simple regression model with the $y_i$ as responses and the $s_i$ as predictor could do pretty well, possibly going to a quadratic or cubic fit if the straight line does not fit well.
If $n$ (my notation) is fairly small as you say, then you may not get anything definitive here. But I'm supposing that you meant it is small in terms of compiling a table of frequencies.
