I am trying to compare the performance of two devices designed to cool anaesthetised patients. This is a substudy analysis and the patient allocation between the groups is not even. Once cooling was started, they were cooled to 33 degrees before being re-warmed. Temperature measurements were taken once every hour. Both devices work on a negative feedback principle and I am investigating the device's ability to maintain patient temperature around the target temperature (33 degrees) for the set time period (28 hours). Below is plotted the mean time course of the groups of patients with the standard deviation. As demonstrated, the standard deviation is less for the IV33 group once cooling is established, although there were fewer patients in this group.


What I am trying to show is that there is a statistically significant difference in the variance once cooling is established. To do that I have applied Levene's Test to the results from each hour which give me the following results:

Hour - p value
00 - 0.8949
01 - 0.8954
02 - 0.2522
03 - 0.01618 
04 - 0.03234 
05 - 0.004928 
06 - 0.000227 
07 - 0.0001289
08 - 0.0002498 
09 - 0.001403 
10 - 0.001158 
11 - 0.0001553 
12 - 0.01084 
13 - 0.0003181
14 - 0.001402 
15 - 0.005558 
16 - 0.01849 
17 - 0.001601 
18 - 0.003469 
19 - 0.01291 
20 - 0.02297 
21 - 0.09245
22 - 0.02421 
23 - 0.03829 
24 - 0.03653 
25 - 0.05466
26 - 0.1282
27 - 0.1982
28 - 0.3297

As you can see, these correlate well with the graph, starting to become significant as soon as cooling takes effect. However, I am obviously applying the same test multiple times. Using a standard Bonferroni correction I would need to use a p value of around 0.0014 (36 hours) which I think is overly conservative as Bonferroni aims at some negative correlation-type dependence whereas my results are rather positively correlated.

Is this correct and if so, how can I calculate a more appropriate p value that takes into account the multiple testing?

  • $\begingroup$ This is a cool example where a test for equality of variances is really needed. $\endgroup$ – Horst Grünbusch Aug 22 '14 at 21:07

You can take a Westfall-Young type approach. Estimate the correlation matrices R1 and R2 of each group. Then draw some 10000 (normally distributed, if you believe in it) random samples with these correlation matrices similar to your original data; just if they were under $H_0$, so equal variances but estimated covariances. Calculate each time all the 36 Levene tests. Save the minimum $p$-values of all tests. At the end, take a empirical $\alpha$ quantile of your 10000 minimum $p$-values. This is the local $\alpha$ that controls the FWER. It will be somewhere between $\frac{\alpha}{36}$ and $\alpha$.

Some brief (untested) R-code to illustrate it: library(mvtnorm) library(lawstat) for (i in 1:10000){ x1 <- rmvnorm(n1,sigma=R1) x2 <- rmvnorm(n2,sigma=R2) for (j in 1:36){ pvali[j] <- levene.test(rbind(x1[,j],x2[,j]), as.factor(rep(1,n1),rep(2,n2)))$p.value } minpval[i] <- min(pvali) } localalpha <- quantile(minpval,probs=alpha)


First of all, you actually do achieve your Bonferroni p-value, so this correction won't prevent you from declaring significance.

As an alternative, consider examining your data for time periods 8 - 20 only. It looks like the mean of both devices are essentially the same there, and there appears to be limited skewness, so we an assume that the distribution of body temperatures during this time period are stationary and have approximately the same median temperature value.

Therefore, we can focus a test on the differences in scale. I would recommend applying the Ansari-Bradley test of differences in scale to the data from time periods 8 - 20. You can perform a two-tailed hypothesis test. The R-code I linked to also gives you a confidence interval for the ratio of the variances, so you can see how practically significant things are.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.