# Multiple comparisons when investigating consistency of a device

I am trying to compare the performance of two devices designed to cool anaesthetised patients. This is a substudy analysis and the patient allocation between the groups is not even. Once cooling was started, they were cooled to 33 degrees before being re-warmed. Temperature measurements were taken once every hour. Both devices work on a negative feedback principle and I am investigating the device's ability to maintain patient temperature around the target temperature (33 degrees) for the set time period (28 hours). Below is plotted the mean time course of the groups of patients with the standard deviation. As demonstrated, the standard deviation is less for the IV33 group once cooling is established, although there were fewer patients in this group. What I am trying to show is that there is a statistically significant difference in the variance once cooling is established. To do that I have applied Levene's Test to the results from each hour which give me the following results:

Hour - p value
00 - 0.8949
01 - 0.8954
02 - 0.2522
03 - 0.01618
04 - 0.03234
05 - 0.004928
06 - 0.000227
07 - 0.0001289
08 - 0.0002498
09 - 0.001403
10 - 0.001158
11 - 0.0001553
12 - 0.01084
13 - 0.0003181
14 - 0.001402
15 - 0.005558
16 - 0.01849
17 - 0.001601
18 - 0.003469
19 - 0.01291
20 - 0.02297
21 - 0.09245
22 - 0.02421
23 - 0.03829
24 - 0.03653
25 - 0.05466
26 - 0.1282
27 - 0.1982
28 - 0.3297


As you can see, these correlate well with the graph, starting to become significant as soon as cooling takes effect. However, I am obviously applying the same test multiple times. Using a standard Bonferroni correction I would need to use a p value of around 0.0014 (36 hours) which I think is overly conservative as Bonferroni aims at some negative correlation-type dependence whereas my results are rather positively correlated.

Is this correct and if so, how can I calculate a more appropriate p value that takes into account the multiple testing?

• This is a cool example where a test for equality of variances is really needed. – Horst Grünbusch Aug 22 '14 at 21:07

You can take a Westfall-Young type approach. Estimate the correlation matrices R1 and R2 of each group. Then draw some 10000 (normally distributed, if you believe in it) random samples with these correlation matrices similar to your original data; just if they were under $H_0$, so equal variances but estimated covariances. Calculate each time all the 36 Levene tests. Save the minimum $p$-values of all tests. At the end, take a empirical $\alpha$ quantile of your 10000 minimum $p$-values. This is the local $\alpha$ that controls the FWER. It will be somewhere between $\frac{\alpha}{36}$ and $\alpha$.
Some brief (untested) R-code to illustrate it:  library(mvtnorm) library(lawstat) for (i in 1:10000){ x1 <- rmvnorm(n1,sigma=R1) x2 <- rmvnorm(n2,sigma=R2) for (j in 1:36){ pvali[j] <- levene.test(rbind(x1[,j],x2[,j]), as.factor(rep(1,n1),rep(2,n2)))\$p.value } minpval[i] <- min(pvali) } localalpha <- quantile(minpval,probs=alpha)