Chi squared or ROC for count data?

Question

The scenario: 54 participants partake in an experiment containing 4 trials (each participant does each trial). In each trial, the participant is presented with a random instance of a categorical stimulus from one of 6 categories (a-f), and provides a categorical response (a-f). The aim of the game is to identify the correct category (i.e. give a response of a to a stimulus of a). There are therefore 216 tests in total and the study is balanced such that each category appears 36 times as a stimulus.

I want to study the effect of stimulus category on the response (i.e. do people perform better or worse for certain categories). To do this I have tried the following 2 approaches:

1) Chi squared goodness of fit, by category. For each category (i.e. a), there are $216/6 = 36$ tests where the 'correct' response is the category (positives) and $216-36 = 180$ tests where the correct response is not the category (negatives). I calculate the expected (chance) values as follows:

• true positives: $tp = 36/6 = 6$

• false negatives: $fn = (216-36)/6 = 30$

• false positives: $fp = 36/6*5 = 30$

• true negatives: $tn = (216-36)/6*5 = 150$

2) Position in terms of ROC space. This is calculated as $tpr/fpr$, where:

• true positive rate: $tpr = tp/36$
• false positive rate: $fpr = fp/180$

Now, it is clear from the above that the expected (chance) values from approach 1 will yield a position in the ROC space that is on the random line ($y=x$).

However, when I plug all the actual data (including information about the negatives) into a chi squared goodness of fit I get very different results. For example, given a category where $tp = 11$ and $fp = 51$, the position in ROC space is close to the random line ($x = 51/180 = 0.28$ and $y = 11/36 = 0.3$), indicating that the results are close to chance. Yet when these numbers are plugged into a chi squared goodness of fit test (I'm using scipy.stats.chisquare in Python), with expected values of [6,30,30,150] and actual values of [11,25,51,129], I get a very low p-value $(<0.01)$. I understand that this difference is due to the information about negatives being included.

Questions

1. Which (if either) of these tests is most suitable for my scenario?
2. Is there a more suitable test that I have not considered?
3. If I want to present the results in terms of ROC, what is a suitable statistical test for comparing the ROC values to an expected (chance) value? In this case, would it be more suitable to use the Chi sq. but with only the $tpr$ and $fpr$ values?

Answer 1

I would try a simpler approach:

1. For each row, find the success rate. I.e. find $P_{aa}, ..., P_{ff}$, where $P_{aa}$ means the probability of selecting category $a$ after being shown $a$
2. Conduct a z-test on the proportions, where you test $H_0: P_{aa} = 1/6$ vs. $H_1: P_{aa}>1/6$. I.e. we are conducting a test against random selection
3. Apply the Bonferroni correction for conducting multiple hypothesis tests. I.e. compare the p-value from step 2 to the usual significance level divided by the number of tests run, $\alpha \mapsto \alpha = 0.05 / 6$