# How can I run multiple tests in a valid and random fashion?

I hope this is the correct place to ask this question. I'm currently doing research where I'm comparing two groups on their EEG frequency spectra. I wasn't too sure how to approach this.

Data:

I have the power values (dB) of multiple frequencies per subject, ranging from 0.6 to 30, in steps of 0.2, thus 148 data points.

Problem:

In general, I would like to know where my two groups (red and black) differ on the power spectrum. I could run a t-test on every 148 bins, but I will surely run into several problems like multiple comparisons.

Question:

Which statistical test would be most appropriate here? Specific types of non-parametric tests? Is there some form of random testing that would be fitting here? Bootstrapping? How would that work? And would you in addition have any tips on how to implement this test (e.g. in R), even though that is out of the scope of this forum.

• one option would be to use a regression model and identify the coefficient differences, so you need to find a sutable model for your power spectrum. – seanv507 Jul 10 '18 at 14:27

I've came up with a solution myself, so maybe this helps someone in the future. A valid way to deal with this kind of data can be permutation testing (see here).

To quote from the article:

The nonparametric statistical test is performed in the following way:

(1) Collect the trials of the two experimental conditions in a single set.

(2) Randomly draw as many trials from this combined data set as there were trials in condition 1 and place those trials into subset 1. Place the remaining trials in subset 2. The result of this procedure is called a random partition.

(3) Calculate the test statistic on this random partition.

(4) Repeat steps 2 and 3 a large number of times and construct a histogram of the test statistics.

(5) From the test statistic that was actually observed and the histogram in step 4, calculate the proportion of random partitions that resulted in a larger test statistic than the observed one. This proportion is called the p-value.

(6) If the p-value is smaller than the critical alpha-level (typically, 0.05), then conclude that the data in the two experimental conditions are significantly different.

Based on a tutorial on youtube by Jonne Sälevä (2015), I created the following code, up to step 5 from above:

## Loop over all the columns of interest
for(i in range of columns){

## Split the data
group1_data <- perm_data[perm_data$Group == "group1",] group2_data <- perm_data[perm_data$Group == "group2",]

## Make separate lists of group and dependent variable data
combined_groups <- c(group1_data$Group, group2_data$Group)
combined_x      <- c(group1_data$x, group2_data$x)

## Number of simulations
nsims <- 10000

## The observed difference in means
diff_obs <- mean(group1_data$x) - mean(group2_data$x)
## Initialize a variable to store the differences
diffs <- rep(NA, nsims)

## Permutation for loop
for(k in 1:nsims){
shuffled_labels <- sample(combined_groups, replace = FALSE)
diffs[k] <- mean(combined_x[shuffled_labels == 1]) - mean(combined_x[shuffled_labels == 2])
}

## p-value = (number of more extreme differences than diff_obs/nsims)
length(diffs[abs(diffs)>= abs(diff_obs)])/nsims
perm_pvalue[index] <- mean(abs(diffs) > abs(diff_obs))

}

• I agree that the optimal strategy here is a permutation test, especially since the power spectrum data is unlikely to be normally distributed (although a log transform of your data may get you closer). However there are two serious issues with above suggestion. The first is that you have a lot of dependencies in your data that you should taken into account when determining the significance (e.g. if neighbouring frequencies show the same pattern, this should strengthen your belief). The other is that the difference in means alone is unlikely to be a good indicator of difference. – Mensen Oct 10 '18 at 13:12
• – Mensen Oct 10 '18 at 13:13