The scenario: 54 participants partake in an experiment containing 4 trials (each participant does each trial). In each trial, the participant is presented with a random instance of a categorical stimulus from one of 6 categories (a-f), and provides a categorical response (a-f). The aim of the game is to identify the correct category (i.e. give a response of a to a stimulus of a). There are therefore 216 tests in total and the study is balanced such that each category appears 36 times as a stimulus.

I want to study the effect of stimulus category on the response (i.e. do people perform better or worse for certain categories). To do this I have tried the following 2 approaches:

1) Chi squared goodness of fit, by category. For each category (i.e. a), there are $216/6 = 36$ tests where the 'correct' response is the category (positives) and $216-36 = 180$ tests where the correct response is not the category (negatives). I calculate the expected (chance) values as follows:

  • true positives: $tp = 36/6 = 6$
  • false negatives: $fn = (216-36)/6 = 30$

  • false positives: $fp = 36/6*5 = 30$

  • true negatives: $tn = (216-36)/6*5 = 150$

2) Position in terms of ROC space. This is calculated as $tpr/fpr$, where:

  • true positive rate: $tpr = tp/36$
  • false positive rate: $fpr = fp/180$

Now, it is clear from the above that the expected (chance) values from approach 1 will yield a position in the ROC space that is on the random line ($y=x$).

However, when I plug all the actual data (including information about the negatives) into a chi squared goodness of fit I get very different results. For example, given a category where $tp = 11$ and $fp = 51$, the position in ROC space is close to the random line ($x = 51/180 = 0.28$ and $y = 11/36 = 0.3$), indicating that the results are close to chance. Yet when these numbers are plugged into a chi squared goodness of fit test (I'm using scipy.stats.chisquare in Python), with expected values of [6,30,30,150] and actual values of [11,25,51,129], I get a very low p-value $(<0.01)$. I understand that this difference is due to the information about negatives being included.


  1. Which (if either) of these tests is most suitable for my scenario?
  2. Is there a more suitable test that I have not considered?
  3. If I want to present the results in terms of ROC, what is a suitable statistical test for comparing the ROC values to an expected (chance) value? In this case, would it be more suitable to use the Chi sq. but with only the $tpr$ and $fpr$ values?
  • $\begingroup$ I'm surprised this was flagged as off-topic--it seems to be precisely on topic, given that it's about choosing a statistical test and interpreting it! $\endgroup$ Commented Feb 13, 2017 at 17:19
  • $\begingroup$ I'm missing why you would think of ROC at all in this context. $\endgroup$ Commented Jan 24, 2019 at 13:37
  • $\begingroup$ @FrankHarrell I am (well was - this was a long time ago!) only interested in whether people identify the correct category or not (i.e. I don't care about the type of misclassification). In this case I don't see why ROC wouldn't be suitable - could you explain why? I'm aware of some discussion around whether it's suitable to 'group' multi-class results into binary data in this manner. Is this the reasoning behind your comment? $\endgroup$
    – tribalsoul
    Commented Jan 29, 2019 at 9:39
  • $\begingroup$ I just don't see what made you think of ROC analysis. Seems to be a category membership probability question. $\endgroup$ Commented Jan 29, 2019 at 13:13
  • $\begingroup$ I guess it's a question of background - mine is in signal processing and machine learning, before statistics, so it just seemed like an obvious way to assess classification accuracy :) $\endgroup$
    – tribalsoul
    Commented Jan 30, 2019 at 14:11

1 Answer 1


I would try a simpler approach:

  1. For each row, find the success rate. I.e. find $P_{aa}, ..., P_{ff}$, where $P_{aa}$ means the probability of selecting category $a$ after being shown $a$
  2. Conduct a z-test on the proportions, where you test $H_0: P_{aa} = 1/6$ vs. $H_1: P_{aa}>1/6$. I.e. we are conducting a test against random selection
  3. Apply the Bonferroni correction for conducting multiple hypothesis tests. I.e. compare the p-value from step 2 to the usual significance level divided by the number of tests run, $\alpha \mapsto \alpha = 0.05 / 6$
  • $\begingroup$ This seems like a suitable approach as it actually yields results that are similar to the points in ROC space (i.e. greater p-values are further from the random line). But I have one concern - it is my understanding that the z test is parametric. I initially performed a Chi squared test for independence on the whole 6*6 contingency table, and then tried the 2 above approaches as post-hocs. Does it make sense to follow up a (non-parametric) test on the whole data with a parametric test as a post-hoc? $\endgroup$
    – tribalsoul
    Commented Feb 15, 2017 at 10:14
  • $\begingroup$ @tribalsoul: Yep, it's absolutely fine to follow up a non-parametric test with a parametric one. The big 'no-no' with conducting too many tests is running so many different ones until you find the answer that you want. But I don't think you are running into that problem. $\endgroup$
    – user64106
    Commented Feb 15, 2017 at 10:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.