# Is it wise to use permutation test with many electrodes and many conditions?

I have data from an EEG (electrophysiological data) experiment where we have more than 10 electrodes recording brain activity (dependent variable) simultaneously. The design is a within subject design with four conditions (independent variables) presented to each subject. I collect EEG amplitudes as dependent variables. For each condition, subjects saw 40 trials.

We are interested in finding which electrodes show statistically significant differences between some of the pairwise comparisons between these four conditions (i.e., A-B, A-C, A-D or A-B, B-C, C-D). There are at least three comparisons like that we are interested in.

I would like to apply permutation or another nanparametric test in order to achieve correction for multiple comparisons for both i) condition comparisons as well as ii) across electrodes. Because I take each electrode as a separate measure and I believe I should correct for family-wise error. One thing to note is that electrode activities may not be fully independent from each other due to the nature of the voltage distribution across scalp (see below).

Many permutation examples use an independent variable with two conditions, and they permute these conditions for each subject for each electrode, and take the maximum t value across electrodes in each permutation, repeating this process 1000 times, and get a tmax distribution, then finding the 95th percentile value as a threshold. This way they control for false discovery rate across electrodes. Some other methods use cluster based method, where they use the spatial proximity of the electrodes and find the minimum electrode cluster size as a threshold. This method considers dependence between electrodes, but overlooks the small but significant clusters. But, all these methods assume simple two condition cases, swapping condition values for permutation test.

My question is: Which nonparametric method would be wise to apply in my situation where I have four within subject conditions as well as many electrodes that these measures are collected?

Thanks

• This isn't very clear to me. Can you say more about your situation, your data, & your goals? What are the "tmax" & "cluster based" algorithms? What given examples are you referring to? Etc. – gung - Reinstate Monica Jan 22 '15 at 3:36
• I understand your question well since I wrote my thesis about it. Please see my paper and my associated toolbox created for matlab for your answer, or contact me directly with any questions. I'm happy to help since it will mean people are doing EEG analysis the right way. Briefly, a threshold free cluster enhancement with max permutation is ideal, and for multiple conditions using the F values from the appropriate ANOVA as your summary measure. – Mensen Jan 22 '15 at 16:07

My PhD thesis was on precisely the topic of how to best test for significant differences in EEG and I faced the same questions.

I found the optimal method is to use a mass-univariate test for each electrode and time/frequency point independently. This may be a t-test, or ANOVA (as in your case a repeated measures ANOVA), or even simply the mean differences, as long as you can justify that the test is valid for your hypothesis.

Then, precisely because there is a high correlation between neighbouring electrodes, and neighbouring time points, the 'trick' is to use the neighbourhood to enhance your original uni-variate measure (e.g. t-values), using threshold-free cluster-enhancement. Essentially this looks at the intensity of the test statistic and scales this value according to the corresponding strength of neighbouring values.

See my NeuroImage paper for details. There I show how the method is logically superior to existing methods, but also use simulations to show precisely its sensitivity and specificity against the methods you already mentioned as well as Statistical Parametric Mapping and Global Field Power.

I've created a user-friendly matlab toolbox to quickly and easily use the method. All you need is your data and the spatial locations of your electrodes and you're ready to go.

You can find the toolbox on Github here.

Feel free to contact me with any questions regarding the method.

• thanks for your answer. I am new to this portal, and was trying to reply to your message. I looked at you paper and found it very relevant. I wonder whether there is any method in the literature using F statistic for nonparametric approach towards multiple comparison across electrodes? In your approach, can I use F values? Thanks, – eegguru Jan 25 '15 at 2:37
• @eegguru, here is a paper actually using F-values in the case of 3 group by 2 condition experiment and the TFCE permutation method successfully.The idea is that instead of using the t-values, just get the F-values for the parameter of interest. There are options for repeated measures and mixed measures in the toolbox but they are limited to 2 factors at the moment since there are some nice statistical shortcuts which makes it fast enough for permutation. – Mensen Jan 25 '15 at 15:09
• Do you know any EEG papers where there are multiple conditions and more than two regions are examined with thresholding method (cluster or max F)? – eegguru Jan 29 '15 at 13:51
• @eegguru, the paper linked to above has multiple groups of participants, which statistically is the same as independent conditions. Moreover, in the paper above, all the time points and channels are examined using the TFCE approach (where are you finding two regions?). Your case with a single factor with 4 levels is more basic than the one described above, so what exactly are you looking for with more references? What about the approach is unclear? – Mensen Jan 29 '15 at 14:28
• For TFCE, you need to use a threshold to select the surviving clusters. In your paper, you choose alpha .05 for the main effects, and .2 for the interactions -but see Fig3,.05. Then you end up,say, 3 clusters surviving. Then, you did ANCOVA on those clusters. My question is: Don't you need to adjust the alpha in the post-hoc tests due to the fact that you will run these post-hoc comparisons on more than one cluster? e.g. If you had only two conditions to compare in the second level of analysis and run t-tests, wouldn't you lower alpha due to multiple tests run across all the clusters? – eegguru Jan 29 '15 at 20:13