I have $3$ groups of patients: baseline, test 1 and test 2. Their sample sizes are $N_0, N_1, N_2$, respectively. The number of observed positive patients are $n_0, n_1, n_2$, respectively. I want to construct the $(1-\alpha)\%$ confidence intervals (CI) for the pairwise changes: $\frac{n_1}{n_0} - 1, \frac{n_2}{n_0} - 1, \frac{n_1}{n_2} - 1$.

My questions are:

  1. Am I in the case of multiple comparison test? that is, for each change, I need to calculate $(1-\frac{\alpha}{3})\%$ CI, instead of $(1-\alpha)\%$ CI (suppose Bonferroni correction is used)
  2. How to calculate these intervals? I tried to model the number of positive patients by binominal distribution, then approximate them by normal distribution (assume the sample size is large enough), then I came up with the ratio of $2$ non-centered gaussians, which turns out to have a very complicated law.

1 Answer 1

  1. For three comparisons, you might not get into so much trouble with Bonferroni, but, as another member once put it, the trouble with Bonferroni isn’t that it’s conservative (unnecessarily wide confidence intervals); the trouble is that Bonferroni is ridiculously conservative. Perhaps consider other methods, such as Bonferroni-Holm.

But you’re on the right track of making an adjustment for your multiple tests.

  1. A typical way of calculating confidence intervals when you don’t know how is to use bootstrap resembling. The gist is that you sample, WITH REPLACEMENT, from the original distribution, to approximate the sampling distribution. There is lots of good stuff out there about bootstrap. I remember liking Professor Knudson’s videos on the topic, and there are a bunch of extensions of vanilla bootstrap methods.

Knudson: https://youtube.com/watch?v=STGGniMV0jg

  • $\begingroup$ thanks for your answer! Boostrap is definitely a very good idea to apply here (I learned about it but clearly failed to realise that it can be used here). And the reason we need the adjustment for multiple tests is because the CI use inter-dependent datasets, is that right? $\endgroup$
    – SiXUlm
    Commented Jun 17, 2020 at 12:08
  • $\begingroup$ It’s because of the XKCD with the jellybeans: xkcd.com/882. $\endgroup$
    – Dave
    Commented Jun 17, 2020 at 12:21
  • $\begingroup$ Nice, oldy but goldy example :) $\endgroup$
    – SiXUlm
    Commented Jun 17, 2020 at 15:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.