How to compute mean and standard deviation of a set of values each of which is itself a mean and has standard deviation?

My data looks like the following:

id        x - y (in m)
-----------------------
1         n 100   m 3.09   s 1.1
2         n 101   m 4.56   s 1.2
3         n 100   m 2.8    s 0.008
4         n 102   m 4.8    s 0.3


The value after n indicates the number of samples, the value after m indicates the mean of those number of samples with a standard deviation given by the value after s. Basically, the data in column "x" has direct summary of differences between two types of distance data (x and y) for a minute (~100 samples). I would like to compute mean of the "x" column and would like to know how to go about it. To compute mean, I just need to calculate weighted mean. But how do I propagate errors (deal with standard deviations) in this case?

• I can't understand what your data is. Commented Jul 7, 2017 at 3:52
• @MichaelChernick: Basically, the data in column "x" has direct summary of differences between two types of distance data (x and y) for a minute (~100 samples). I would like to compute the mean of this column and would like to know what to do with standard deviations. Commented Jul 7, 2017 at 4:00

The key idea here is that you are describing a mixture distribution. There are a couple of ways of combining multiple sets of data: you could be interested in the sum (difference, ratio, etc.) of multiple distributions or in the concatenation of them. Consider this example:

A:       1  2  3
B:       4  5  6
A+B:     5  7  9
{A, B}:  1  2  3  4  5  6


A+B yields the distribution of the sums of the corresponding elements, whereas {A, B} simply puts all the values into a single collection. I gather you want to determine the mean and SD that you would have computed if you had used the standard formulas on the complete set of 403 data instead of only having access to summary statistics on each of the four subsets.

The concatenation of the four subsets is a mixture distribution. So we need to use the formulas for that. They are:

\begin{align} \bar x_. &= \frac{\sum_i n_i \bar x_i}{\sum_i n_i} \\[5pt] s_{.}^2 &= \frac{\sum_i n_i ((\bar x_i - \bar x_.)^2 + s^2_i)}{\sum_i n_i} \end{align}

In these formulas, $i$ indexes the subsets, and the dot ($.$) indicates the value for the complete set. The second formula pertains to the variance, but you can take the square root to get the SD.

There is a tricky aspect to the formula for the variance, however. It is the formula for the maximum likelihood estimate of the variance, sometimes called the "population" variance, whereas your SDs are presumably what are sometimes called "sample" SDs. That is, I suspect the SDs you list were calculated using Bessel's correction, and that, moreover, you want the sample SD for the mixture. To get that, we need to make two adjustments to the above formula: first, 'undo' Bessel's correction, and then apply it at the end.

\begin{align} \color{red}{\tilde s^2_i} &\color{red}{= s^2_i \frac{(n_i - 1)}{n_i}} \\[5pt] s_{.}^2 &= \frac{\sum_i n_i ((\bar x_i - \bar x_.)^2 + \color{red}{\tilde{\color{black}{s}}}^2_i)}{(\sum_i n_i) \color{red}{-1}} \end{align}

We can now see how this works with the example data above (the following is coded in R, but is intended to be self-explanatory even for those who do not use R):

mix.mean = function(x.bars, ns){
sum(ns*x.bars) / sum(ns)
}
mix.var = function(x.bars, mix.mean, s2s, ns){
s2s = s2s * (ns-1)/ns  # this 'takes out' Besel's correction
sum(ns*((x.bars - mix.mean)^2 + s2s)) / (sum(ns)-1)
}

A = c(1, 2, 3)
B = c(4, 5, 6)
c(length(A), length(B))  # [1] 3 3
c(  mean(A),   mean(B))  # [1] 2 5
c(   var(A),    var(B))  # [1] 1 1
mix.mean(x.bars=c(mean(A), mean(B)), ns=c(length(A), length(B)))  # [1] 3.5
mean(c(A, B))                                                     # [1] 3.5

mix.var(x.bars=c(mean(A), mean(B)), mix.mean=3.5,
s2s=c(var(A), var(B)), ns=c(length(A), length(B)))  # [1] 3.5
var(c(A, B))                                                # [1] 3.5

• Yes, it's a mixture of Gaussians. You got my problem statement correct. But I wonder how and why the formula for sd given in the previous answer fails. Can you please elaborate on this? Is it because it doesn't take into account the standard deviation of the resulting mixture in addition to the sd of the individual components of the mixture? Commented Jul 7, 2017 at 20:57
• @Gingerbread, that version converts each s2 to a sum of squares, so I don't see that as a problem. OTOH, it treats the subsets as though they all have equal means to infinite decimal places. That is likely to be a problem, but otherwise, it would be OK, I believe. Commented Jul 7, 2017 at 23:50

Let's say your data has the structure $$\begin{array}{cccc} i & n_i & \bar x_i & s_i \\ \hline 1 & 100 & 3.09 & 1.1 \\ 2 & 101 & 4.56 & 1.2 \\ \vdots \end{array}$$ where $i$ represents an index of summarized observations, $n_i$ represents the sample size of the $i^{\rm th}$ observation $\boldsymbol x_i = (x_{i1}, x_{i2}, \ldots, x_{in_i})$, $\bar x_i = (x_{i1} + \cdots + x_{in_i})/n_i$ is the sample mean of the $i^{\rm th}$ observation, and $s_i$ is the sample standard deviation of $\boldsymbol x_i$. Then if each $x_{ij}$ is to have equal weight and the observations are assumed to be independent realizations, the pooled variance $s$ can be computed as the weighted sum of the individual observed variances; e.g., $$s^2 = \frac{1}{n - 1} \sum_{i=1}^m (n_i - 1) s_i^2,$$ where there are $m$ observations (rows) in your data, and $n = n_1 + \cdots + n_m$ is the total sample size.

• The pooled variance does not seem to correspond to anything requested or described in the question.
– whuber
Commented Jul 7, 2017 at 22:08