# Multiple Comparison correction for temporally correlated tests

My data consists of N~200 cells. For each cell I am seeing the response to a stimulus for the next 300ms, in bins of 5ms (150bins total). In order to get confidence bounds for each bin on whether a response is significant, I perform a shuffling/bootstrapping test. This gives me graphs which look like those in figure below for each cell: [Figure shows 4 cell responces (blue lines) w/ 95% confidence bounds (grey).]

If cells responses go above bounds for even 1 bin, they are considered significant and have a blue background. I understand that essentially I am doing 150 comparisons here, but a Bonferroni correction here seems incorrect due to the tests being temporally correlated.

Any suggestions on how to correctly ascertain whether the the cells have a significant response?

According to http://www.fil.ion.ucl.ac.uk/spm/doc/papers/NicholsHolmes.pdf, you can correct the p-values for multiple comparisons by comparing the lengths of the regions out of the confidence interval to ones obtained by permutation. This is presented for 3D signal. In your case, it is about 1D signal but I see no restriction to the dimension of the space. More precisely, the method is the following:

1. fix a suprathreshold value $\alpha$ typically 0.05,

2. build the permutation distribution of the maximal suprathreshold group size (shuffling the p-values over the line and computing the size of the largest group of points with p-values greater than $\alpha$ for $N$ generated permutation)

3. set the critical suprathreshold group size as the [$\alpha$ × $N$]+1 th largest value over the sampling distribution.

4. then a response is significant if a region of $\alpha$ significant point of size larger than the suprathreshold group size exists.

• thank you! which section of the paper did you get this from? – DankMasterDan Feb 16 '16 at 18:10
• @DankMasterDan page 7 subsection Suprathreshold cluster tests. This is presented for 3D signal. In your case, it is about 1D signal but I see no restriction to the dimension of the space. Hope it helps. – peuhp Feb 17 '16 at 8:56
• Thnx, just the course I was looking for ! – DankMasterDan Feb 17 '16 at 18:07
• Great, this method is pretty classic in functional mri (the paper is almost cited 3000 times). Happy to help – peuhp Feb 17 '16 at 19:05