It is common practice in neural decoding studies to compare the decoding accuracy on a data set with the accuracy on the label-shuffled data set, and then use a statistical test to determine whether decoding performance on the unshuffled data is significantly higher.

Suppose a dataset consists of $n$ accuracy scores on $n$ sessions or trials, and $n$ accuracies on the corresponding shuffled data. My question is what would be the most appropriate statistical test to address the difference between the two population means.

My confusion arises mainly because some studies use a Wilcoxon rank sum test (Mann-Whitney U-test), for example this paper, which is a test for independent samples. Others, see for example this article, use a paired t-test, which as the name suggests is tailored to paired observations.

I would say that a score computed on unshuffled and shuffled data is a dependent observation, as there is a one-to-one correspondence between points in each pair, and thus a Wilcoxon rank sum would be inappropriate (the Wilcoxon signed rank test would be a paired-samples nonparametric alternative). I have also noticed that the Wilcoxon rank sum produces lower p-value than the paired t test.

  • 1
    $\begingroup$ Are you referring to a permutation test maybe? One where you 'shuffle' the values or labels and check how often across all 'shuffles' a difference from the null-hypothesis is found? $\endgroup$ – IWS Apr 5 '17 at 12:47
  • $\begingroup$ Not exactly. A permutation test would be where you take the difference between the means of population A (accuracy on unshuffled data) and population B (accuracy on shuffled data) as a test statistic and compare it to the null distribution in which you randomize the 'shuffled' and 'unshuffled' labels. I am not sure whether this test is applicable to paired data. In fact in the first paper I mention they use both Wilcoxon rank sum and this permutation test. $\endgroup$ – Pietro Marchesi Apr 5 '17 at 13:26
  • $\begingroup$ I'm not sure whether I agree with your explanation of a permutation test, but maybe you could add some sample data to your question to make clear what kind of test (permutation or something else) you are looking for. $\endgroup$ – IWS Apr 5 '17 at 13:29
  • $\begingroup$ I am referring to this (section on permutation tests). The data is explained in my question: you have n scores (accuracy or other) on n sessions/trials, and n accuracies on the corresponding shuffled data (each session\trial shuffled independently), and you are interested in whether the mean of the unshuffled population is significantly higher. $\endgroup$ – Pietro Marchesi Apr 5 '17 at 13:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.