The scenario: 54 participants partake in an experiment containing 4 trials (each participant does each trial). In each trial, the participant is presented with a random instance of a categorical stimulus from one of 6 categories (
f), and provides a categorical response (
f). The aim of the game is to identify the correct category (i.e. give a response of
a to a stimulus of
a). There are therefore 216 tests in total and the study is balanced such that each category appears 36 times as a stimulus.
I want to study the effect of stimulus category on the response (i.e. do people perform better or worse for certain categories). To do this I have tried the following 2 approaches:
1) Chi squared goodness of fit, by category. For each category (i.e.
a), there are $216/6 = 36$ tests where the 'correct' response is the category (positives) and $216-36 = 180$ tests where the correct response is not the category (negatives). I calculate the expected (chance) values as follows:
- true positives: $tp = 36/6 = 6$
false negatives: $fn = (216-36)/6 = 30$
false positives: $fp = 36/6*5 = 30$
- true negatives: $tn = (216-36)/6*5 = 150$
2) Position in terms of ROC space. This is calculated as $tpr/fpr$, where:
- true positive rate: $tpr = tp/36$
- false positive rate: $fpr = fp/180$
Now, it is clear from the above that the expected (chance) values from approach 1 will yield a position in the ROC space that is on the random line ($y=x$).
However, when I plug all the actual data (including information about the negatives) into a chi squared goodness of fit I get very different results. For example, given a category where $tp = 11$ and $fp = 51$, the position in ROC space is close to the random line ($x = 51/180 = 0.28$ and $y = 11/36 = 0.3$), indicating that the results are close to chance. Yet when these numbers are plugged into a chi squared goodness of fit test (I'm using scipy.stats.chisquare in Python), with expected values of [6,30,30,150] and actual values of [11,25,51,129], I get a very low p-value $(<0.01)$. I understand that this difference is due to the information about negatives being included.
- Which (if either) of these tests is most suitable for my scenario?
- Is there a more suitable test that I have not considered?
- If I want to present the results in terms of ROC, what is a suitable statistical test for comparing the ROC values to an expected (chance) value? In this case, would it be more suitable to use the Chi sq. but with only the $tpr$ and $fpr$ values?