# Chi squared or ROC for count data?

The scenario: 54 participants partake in an experiment containing 4 trials (each participant does each trial). In each trial, the participant is presented with a random instance of a categorical stimulus from one of 6 categories (a-f), and provides a categorical response (a-f). The aim of the game is to identify the correct category (i.e. give a response of a to a stimulus of a). There are therefore 216 tests in total and the study is balanced such that each category appears 36 times as a stimulus.

I want to study the effect of stimulus category on the response (i.e. do people perform better or worse for certain categories). To do this I have tried the following 2 approaches:

1) Chi squared goodness of fit, by category. For each category (i.e. a), there are $216/6 = 36$ tests where the 'correct' response is the category (positives) and $216-36 = 180$ tests where the correct response is not the category (negatives). I calculate the expected (chance) values as follows:

• true positives: $tp = 36/6 = 6$
• false negatives: $fn = (216-36)/6 = 30$

• false positives: $fp = 36/6*5 = 30$

• true negatives: $tn = (216-36)/6*5 = 150$

2) Position in terms of ROC space. This is calculated as $tpr/fpr$, where:

• true positive rate: $tpr = tp/36$
• false positive rate: $fpr = fp/180$

Now, it is clear from the above that the expected (chance) values from approach 1 will yield a position in the ROC space that is on the random line ($y=x$).

However, when I plug all the actual data (including information about the negatives) into a chi squared goodness of fit I get very different results. For example, given a category where $tp = 11$ and $fp = 51$, the position in ROC space is close to the random line ($x = 51/180 = 0.28$ and $y = 11/36 = 0.3$), indicating that the results are close to chance. Yet when these numbers are plugged into a chi squared goodness of fit test (I'm using scipy.stats.chisquare in Python), with expected values of [6,30,30,150] and actual values of [11,25,51,129], I get a very low p-value $(<0.01)$. I understand that this difference is due to the information about negatives being included.

Questions

1. Which (if either) of these tests is most suitable for my scenario?
2. Is there a more suitable test that I have not considered?
3. If I want to present the results in terms of ROC, what is a suitable statistical test for comparing the ROC values to an expected (chance) value? In this case, would it be more suitable to use the Chi sq. but with only the $tpr$ and $fpr$ values?
• I'm surprised this was flagged as off-topic--it seems to be precisely on topic, given that it's about choosing a statistical test and interpreting it! – Matt Krause Feb 13 '17 at 17:19
• I'm missing why you would think of ROC at all in this context. – Frank Harrell Jan 24 at 13:37
• @FrankHarrell I am (well was - this was a long time ago!) only interested in whether people identify the correct category or not (i.e. I don't care about the type of misclassification). In this case I don't see why ROC wouldn't be suitable - could you explain why? I'm aware of some discussion around whether it's suitable to 'group' multi-class results into binary data in this manner. Is this the reasoning behind your comment? – tribalsoul Jan 29 at 9:39
• I just don't see what made you think of ROC analysis. Seems to be a category membership probability question. – Frank Harrell Jan 29 at 13:13
• I guess it's a question of background - mine is in signal processing and machine learning, before statistics, so it just seemed like an obvious way to assess classification accuracy :) – tribalsoul Jan 30 at 14:11

I would try a simpler approach:

1. For each row, find the success rate. I.e. find $P_{aa}, ..., P_{ff}$, where $P_{aa}$ means the probability of selecting category $a$ after being shown $a$
2. Conduct a z-test on the proportions, where you test $H_0: P_{aa} = 1/6$ vs. $H_1: P_{aa}>1/6$. I.e. we are conducting a test against random selection
3. Apply the Bonferroni correction for conducting multiple hypothesis tests. I.e. compare the p-value from step 2 to the usual significance level divided by the number of tests run, $\alpha \mapsto \alpha = 0.05 / 6$
• This seems like a suitable approach as it actually yields results that are similar to the points in ROC space (i.e. greater p-values are further from the random line). But I have one concern - it is my understanding that the z test is parametric. I initially performed a Chi squared test for independence on the whole 6*6 contingency table, and then tried the 2 above approaches as post-hocs. Does it make sense to follow up a (non-parametric) test on the whole data with a parametric test as a post-hoc? – tribalsoul Feb 15 '17 at 10:14
• @tribalsoul: Yep, it's absolutely fine to follow up a non-parametric test with a parametric one. The big 'no-no' with conducting too many tests is running so many different ones until you find the answer that you want. But I don't think you are running into that problem. – user64106 Feb 15 '17 at 10:17