22
$\begingroup$

Imagine that you repeat an experiment three times. In each experiment, you collect triplicate measurements. The triplicates tend to be fairly close together, compared to the differences among the three experimental means. Computing the grand mean is pretty easy. But how can one compute a confidence interval for the grand mean?

Sample data:

Experiment 1: 34, 41, 39

Experiment 2: 45, 51, 52

Experiment 3: 29, 31, 35

Assume that the replicate values within an experiment follow a Gaussian distribution, as does the mean values of each experiment. The SD of variation within an experiment is smaller than the SD among experimental means. Assume also that there is no ordering of the three values in each experiment. The left-to-right order of the three values in each row is entirely arbitrary.

The simple approach is to first compute the mean of each experiment: 38.0, 49.3, and 31.7, and then compute the mean, and its 95% confidence interval, of those three values. Using this method, the grand mean is 39.7 with the 95% confidence interval ranging from 17.4 to 61.9.

The problem with that approach is that it totally ignores the variation among triplicates. I wonder if there isn't a good way to account for that variation.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Not an answer, just an intuitive observation. The CI for the pooled data mean (all nine obs) is $(39.7 \pm 2.13)$, CI based on the means only is $(39.7\pm 12.83)$. Not sure what your CI is doing (typo? 17 not 27, and 51 not 61?), I get $2.98$ for std err of three means, and $4.30$ as $0.975$ quantile of T dist with 2 df. I would think that the CI you seek would lie somewhere in between these two - as you have partial pooling. Could also think in terms of variance formula $V(Y)=E[V(Y|Y_g)]+V[E(Y|Y_g)]$, each CI uses half of the formula $\endgroup$
    – probabilityislogic
    Commented Jun 18, 2011 at 4:58
  • 2
    $\begingroup$ @probabilityislogic: The SEM of the three experiment means is 5.168 (not 2.98 as you wrote), and the confidence interval I gave in the original post (17.4 to 61.9) is correct. The SEM is computed from the SD (8.95) by dividing by the square root of n (square root of 3). You divided by n (3) instead. $\endgroup$
    – Harvey Motulsky
    Commented Jun 19, 2011 at 13:42
  • $\begingroup$ my mistake, should also replace $2.13$ by $6.40$ in the pooled interval (same mistake there) $\endgroup$
    – probabilityislogic
    Commented Jun 20, 2011 at 5:10
  • $\begingroup$ does the following link answers' this? talkstats.com/showthread.php/11554-mean-of-means $\endgroup$
    – user14015
    Commented Sep 11, 2012 at 21:55
  • $\begingroup$ @TST, There appears to be nothing but a link to Wikipedia on Pooled variance. Care to elaborate? $\endgroup$
    – chl
    Commented Sep 12, 2012 at 7:45

4 Answers 4

Reset to default
6
$\begingroup$

There is a natural exact confidence interval for the grandmean in the balanced random one-way ANOVA model $$(y_{ij} \mid \mu_i) \sim_{\text{iid}} {\cal N}(\mu_i, \sigma^2_w), \quad j=1,\ldots,J, \qquad \mu_i \sim_{\text{iid}} {\cal N}(\mu, \sigma^2_b), \quad i=1,\ldots,I.$$ Indeed, it is easy to check that the distribution of the observed means $\bar{y}_{i\bullet}$ is $\bar{y}_{i\bullet} \sim_{\text{iid}} {\cal N}(\mu, \tau^2)$ with $\tau^2=\sigma^2_b+\frac{\sigma^2_w}{J}$, and it is well known that the between sum of squares $SS_b$ has distribution $$SS_b \sim J\tau^2\chi^2_{I-1}$$ and is independent of the overall observed mean $$\bar y_{\bullet\bullet} \sim {\cal N}(\mu, \frac{\tau^2}{I})$$. Thus $$\frac{\bar y_{\bullet\bullet} - \mu}{\frac{1}{\sqrt{I}}\sqrt{\frac{SS_b}{J(I-1)}}}$$ has a Student $t$ distribution with $I-1$ degrees of freedom, wherefrom it is easy to get an exact confidence interval about $\mu$.

Note that this confidence interval is nothing but the classical interval for a Gaussian mean by considering only the group means $\bar{y}_{i\bullet}$ as the observations. Thus the simple approach you mention:

The simple approach is to first compute the mean of each experiment: 38.0, 49.3, and 31.7, and then compute the mean, and its 95% confidence interval, of those three values. Using this method, the grand mean is 39.7 with the 95% confidence interval ranging from 17.4 to 61.9.

is right. And your intuition about the ignored variation:

The problem with that approach is that it totally ignores the variation among triplicates. I wonder if there isn't a good way to account for that variation.

is wrong. I also mention the correctness of such a simplification in https://stats.stackexchange.com/a/72578/8402

Update 12/04/2014

Some details are now written on my blog: Reducing a model to get confidence intervals.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Any help implementing this solution in python? stackoverflow.com/questions/45682437/… $\endgroup$
    – blehman
    Commented Aug 14, 2017 at 20:22
  • $\begingroup$ Note that this answer contains a small but deadly error. The distribution of $SS_b$ is actually just $\tau_2 \chi^2_{I-1}$ (no $J$), so there is no $J$ in the denominator of the $t$-statistic. (I worked through the math, then checked with some simulations.) $\endgroup$
    – Homer White
    Commented Apr 1 at 22:00
7
$\begingroup$

This is a question of estimation within a linear mixed effects model. The problem is that the variance of the grand mean is a weighted sum of two variance components which have to be separately estimated (via an ANOVA of the data). The estimates have different degrees of freedom. Therefore, although one can attempt to construct a confidence interval for the mean using the usual small-sample (Student t) formulas, it is unlikely to attain its nominal coverage because the deviations from the mean will not exactly follow a Student t distribution.

A recent (2010) article by Eva Jarosova, Estimation with the Linear Mixed Effects Model, discusses this issue. (As of 2015 it no longer appears to be available on the Web.) In the context of a "small" dataset (even so, about three times larger than this one), she uses simulation to evaluate two approximate CI calculations (the well-known Satterthwaite approximation and the "Kenward-Roger's method"). Her conclusions include

Simulation study revealed that quality of estimation of covariance parameters and consequently adjustment of confidence intervals in small samples can be quite poor.... A poor estimation may influence not only the true confidence level of conventional intervals but it can also make the adjustment impossible. It is obvious that even for balanced data three types of intervals [conventional, Satterthwaite, K-R] may differ substantially. When a striking difference between the conventional and the adjusted intervals is observed, standard errors of covariance parameter estimates should be checked. On the other hand, when the differences between [the three] types of intervals are small, the adjustment seems to be unnecessary.

In short, a good approach seems to be

  1. Compute a conventional CI by using the estimates of variance components and pretending a t-distribution applies.

  2. Also compute at least one of the adjusted CIs.

  3. If the computations are "close," accept the conventional CI. Otherwise, report that there are insufficient data to produce a reliable CI.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Using the variance components leads to the same confidence interval I computed in the original post. The ANOVA table has a SS between columns of 480.7 with 2 df, which means the MS is 240.3. The SD is sqrt(MSbetween/n) = sqrt(240.3/3) = 8.95, which leads to the same CI I originally posted (17.4 to 61.9). I found it very hard to follow the Jarasova paper you cited, and am not entirely sure it is relevant here (it seems to be about repeated measures designs). ??? $\endgroup$
    – Harvey Motulsky
    Commented Jun 19, 2011 at 14:16
  • $\begingroup$ @Harvey Your description sure sounds like repeated measures to me! I believe the Jarasova paper is spot on. $\endgroup$
    – whuber ♦
    Commented Jun 20, 2011 at 14:13
  • 1
    $\begingroup$ I am thinking of the common situation in labs where the triplicates are simply three different test tubs (or wells). The order of the three as presented in the table is arbitrary. There is no connection or correlation between replicate #2 in the first experiment with replicate #2 in the second or third experiments. Each experiment just has three measurements. So not really repeated measures. Right? $\endgroup$
    – Harvey Motulsky
    Commented Jun 20, 2011 at 15:08
  • $\begingroup$ whuber, there is an exact Student distribution here. See my answer. $\endgroup$
    – Stéphane Laurent
    Commented Oct 12, 2013 at 10:02
  • $\begingroup$ @whuber the link you supply for Eva Jarasova's article is dead and a Google search did not yield anything. Can you correct the reference? $\endgroup$
    – Placidia
    Commented Jan 21, 2015 at 15:52
0
$\begingroup$

You can't have one confidence interval that solves both of your problems. You have to pick one. You can either derive one from a mean square error term of within experiment variance that allows you to say something about how accurately you can estimate the values within experiment or you can do it between and it will be about between experiments. If I just did the former I'd tend to want to plot it around 0 rather than around the grand mean because it doesn't tell you anything about the actual mean value, only about an effect (in this case 0). Or you could just plot both and describe what they do.

You've got a handle on the between one. For the within it's just like calculating the error term in an ANOVA to get an MSE to work with and from there the SE for the CI is just sqrt(MSE/n) (n = 3 in this case).

Improve this answer
$\endgroup$
4
  • $\begingroup$ Actually you can have a credible interval for each mean and for the grand mean. Just use a Bayesian multilevel model. Sometimes this kind of estimate is called partial pooling. The problem is tha small sample, I think. $\endgroup$
    – Manoel Galdino
    Commented Jun 16, 2011 at 20:32
  • $\begingroup$ You could have a confidence interval for each mean and the grand mean too... but they're different things... just like the credible intervals are. I interpreted the question as being about CI's with respect to the within study variance and the between as an aggregate. It all still leaves you with different CI's meaning different things. (I also didn't take the n literally) $\endgroup$
    – John
    Commented Jun 16, 2011 at 21:40
  • 1
    $\begingroup$ In addition, the way I meant can't isn't really "cannot". You could somehow come up with a single equation that calculates one confidence interval for everything. It just wouldn't mean anything sensible. That's what I meant for can't. $\endgroup$
    – John
    Commented Jun 16, 2011 at 21:41
  • $\begingroup$ A few minutes after I wrote my comment I realized that we weren't supposed to take the n literally. But it was to late to edit it =). $\endgroup$
    – Manoel Galdino
    Commented Jun 16, 2011 at 22:15
0
$\begingroup$

I think the CI for grand mean is too wide [17,62] even for the range of original data.

This experiments are VERY common in chemistry. For example, in certification of reference materials you have to pick up some bottles from whole lot in a random way, and you have to carry out replicate analysis on each bottles. How do you calculate the reference value and its uncertainty? There are a lot of way to do it, but the most sofisticated (and correct, I think) is applying meta-analysis or ML (Dersimonian-Laird, Vangel-Rukhin, etc)

What about bootstrap estimates?

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Simulation (10,000 trials with normally distributed main effects and errors) indicates [21, 58] is a symmetric two-sided 95% CI for the mean. $\endgroup$
    – whuber ♦
    Commented Jul 20, 2011 at 19:36
  • $\begingroup$ whuber: I'd be curious to know how you did those simulations. Bootstrapping from the original data? Or truly simulations? If the latter, what value of mean and SD did you use to simulate data?? $\endgroup$
    – Harvey Motulsky
    Commented Aug 14, 2011 at 20:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.