I am running a binary classification over 100 subjects' EEG data, and I want to test the average (over subjects) performance's significance using permutation test. I need to generate a null distribution. Could anyone help me how to produce null distribution?

I have presented two types of sounds to the subjects for 1000ms (1000 time points). So there are two condition in the experiment. For each subject and condition, there are 200 repetitions. So the dimension of data for each subject is: 400 (trials) * 32 (channels) * 1000 (time points). For each subject and at each time point, I have performed a binary classification to discriminate between the two conditions. This gives a classification performance for each time point and each subject. I have averaged this signal over subjects to get the mean classification performance (the dimension is: 1*1000). Now I want to test the significance of the results (decoding between two conditions) at each time point using permutation test.

  • 1
    $\begingroup$ Probably can, but not without the data and a clearer idea what permutation test is contemplated. $\endgroup$ – BruceET May 7 at 6:34
  • 1
    $\begingroup$ Thank you for your reply. I have access to the data. The aim is to test if the classification performance is significant at each time point. I am thinking to put all the trials for all subjects in a matrix and randomly choose a number of trials and put them randomly in two classes, then perform classification on these classes. Will repeating these steps 10000 times generate a reliable null distribution? $\endgroup$ – pinar May 7 at 7:02
  • 1
    $\begingroup$ Maybe show typical results for 2 out of the 100 subjects. How many time points? Comparing each subject with him/herself under 2 different conditions? Comparing subjects with one another? Looking for changes across time?// Exactly what do you mean by 'binary'? // How many permutation tests altogether? // Please edit key info into your Question. Not everyone looks at comments. $\endgroup$ – BruceET May 7 at 7:12
  • $\begingroup$ I could also help you to choose a more informative title. $\endgroup$ – kjetil b halvorsen May 13 at 21:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.