1
$\begingroup$

My data looks like the following:

id        x - y (in m)    
-----------------------      
1         n 100   m 3.09   s 1.1
2         n 101   m 4.56   s 1.2
3         n 100   m 2.8    s 0.008
4         n 102   m 4.8    s 0.3

The value after n indicates the number of samples, the value after m indicates the mean of those number of samples with a standard deviation given by the value after s. Basically, the data in column "x" has direct summary of differences between two types of distance data (x and y) for a minute (~100 samples). I would like to compute mean of the "x" column and would like to know how to go about it. To compute mean, I just need to calculate weighted mean. But how do I propagate errors (deal with standard deviations) in this case?

$\endgroup$
  • $\begingroup$ I can't understand what your data is. $\endgroup$ – Michael Chernick Jul 7 '17 at 3:52
  • $\begingroup$ @MichaelChernick: Basically, the data in column "x" has direct summary of differences between two types of distance data (x and y) for a minute (~100 samples). I would like to compute the mean of this column and would like to know what to do with standard deviations. $\endgroup$ – Gingerbread Jul 7 '17 at 4:00
  • 1
    $\begingroup$ This seems clear enough to be answerable to me, although I acknowledge it is presented rather confusingly. The existence of an upvoted & accepted answer provides some support for this. I'm voting to leave open. $\endgroup$ – gung Jul 7 '17 at 14:49
4
$\begingroup$

The key idea here is that you are describing a mixture distribution. There are a couple of ways of combining multiple sets of data: you could be interested in the sum (difference, ratio, etc.) of multiple distributions or in the concatenation of them. Consider this example:

A:       1  2  3
B:       4  5  6
A+B:     5  7  9
{A, B}:  1  2  3  4  5  6

A+B yields the distribution of the sums of the corresponding elements, whereas {A, B} simply puts all the values into a single collection. I gather you want to determine the mean and SD that you would have computed if you had used the standard formulas on the complete set of 403 data instead of only having access to summary statistics on each of the four subsets.

The concatenation of the four subsets is a mixture distribution. So we need to use the formulas for that. They are:

\begin{align} \bar x_. &= \frac{\sum_i n_i \bar x_i}{\sum_i n_i} \\[5pt] s_{.}^2 &= \frac{\sum_i n_i ((\bar x_i - \bar x_.)^2 + s^2_i)}{\sum_i n_i} \end{align}

In these formulas, $i$ indexes the subsets, and the dot ($.$) indicates the value for the complete set. The second formula pertains to the variance, but you can take the square root to get the SD.

There is a tricky aspect to the formula for the variance, however. It is the formula for the maximum likelihood estimate of the variance, sometimes called the "population" variance, whereas your SDs are presumably what are sometimes called "sample" SDs. That is, I suspect the SDs you list were calculated using Bessel's correction, and that, moreover, you want the sample SD for the mixture. To get that, we need to make two adjustments to the above formula: first, 'undo' Bessel's correction, and then apply it at the end.

\begin{align} \color{red}{\tilde s^2_i} &\color{red}{= s^2_i \frac{(n_i - 1)}{n_i}} \\[5pt] s_{.}^2 &= \frac{\sum_i n_i ((\bar x_i - \bar x_.)^2 + \color{red}{\tilde{\color{black}{s}}}^2_i)}{(\sum_i n_i) \color{red}{-1}} \end{align}

We can now see how this works with the example data above (the following is coded in R, but is intended to be self-explanatory even for those who do not use R):

mix.mean = function(x.bars, ns){
  sum(ns*x.bars) / sum(ns)
}
mix.var = function(x.bars, mix.mean, s2s, ns){
  s2s = s2s * (ns-1)/ns  # this 'takes out' Besel's correction
  sum(ns*((x.bars - mix.mean)^2 + s2s)) / (sum(ns)-1)
}

A = c(1, 2, 3)
B = c(4, 5, 6)
c(length(A), length(B))  # [1] 3 3
c(  mean(A),   mean(B))  # [1] 2 5
c(   var(A),    var(B))  # [1] 1 1
mix.mean(x.bars=c(mean(A), mean(B)), ns=c(length(A), length(B)))  # [1] 3.5
mean(c(A, B))                                                     # [1] 3.5

mix.var(x.bars=c(mean(A), mean(B)), mix.mean=3.5, 
        s2s=c(var(A), var(B)), ns=c(length(A), length(B)))  # [1] 3.5
var(c(A, B))                                                # [1] 3.5
$\endgroup$
  • $\begingroup$ Yes, it's a mixture of Gaussians. You got my problem statement correct. But I wonder how and why the formula for sd given in the previous answer fails. Can you please elaborate on this? Is it because it doesn't take into account the standard deviation of the resulting mixture in addition to the sd of the individual components of the mixture? $\endgroup$ – Gingerbread Jul 7 '17 at 20:57
  • $\begingroup$ @Gingerbread, that version converts each s2 to a sum of squares, so I don't see that as a problem. OTOH, it treats the subsets as though they all have equal means to infinite decimal places. That is likely to be a problem, but otherwise, it would be OK, I believe. $\endgroup$ – gung Jul 7 '17 at 23:50
3
$\begingroup$

Let's say your data has the structure $$\begin{array}{cccc} i & n_i & \bar x_i & s_i \\ \hline 1 & 100 & 3.09 & 1.1 \\ 2 & 101 & 4.56 & 1.2 \\ \vdots \end{array}$$ where $i$ represents an index of summarized observations, $n_i$ represents the sample size of the $i^{\rm th}$ observation $\boldsymbol x_i = (x_{i1}, x_{i2}, \ldots, x_{in_i})$, $\bar x_i = (x_{i1} + \cdots + x_{in_i})/n_i$ is the sample mean of the $i^{\rm th}$ observation, and $s_i$ is the sample standard deviation of $\boldsymbol x_i$. Then if each $x_{ij}$ is to have equal weight and the observations are assumed to be independent realizations, the pooled variance $s$ can be computed as the weighted sum of the individual observed variances; e.g., $$s^2 = \frac{1}{n - 1} \sum_{i=1}^m (n_i - 1) s_i^2,$$ where there are $m$ observations (rows) in your data, and $n = n_1 + \cdots + n_m$ is the total sample size.

$\endgroup$
  • 1
    $\begingroup$ The pooled variance does not seem to correspond to anything requested or described in the question. $\endgroup$ – whuber Jul 7 '17 at 22:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.