I need to calculate the average and standard deviation of a population. However, I only have the information of sample means and standard deviations.

To be more precise, I have a vector of means $\bar{\mathbf{x}}=(\bar{x}_1, \bar{x}_2, \ldots, \bar{x}_N)$ with corresponding vector $\mathbf{n}=(n_1, n_2, \ldots, n_N)$ of sample sizes and a vector of standard deviations $\mathbf{s}=(s_1, s_2, \ldots, s_N)$. I need to compute the overall average and standard deviation.

To obtain the overall mean, I calculated it using the formula $$\bar{x}=\dfrac{\sum_{i=1}^N\bar{x}_in_i}{\sum_{i=1}^Nn_i}.$$ Whereas, for the overall standard deviation, I can compute it from the overall deviation $$Var=\frac{1}{n−1}\left(\sum_{j=1}^N(n_j−1)V_j+\sum_{j=1}^Nn_j(\bar{x}_j−\bar{x})^2\right)\,\;,$$ where $V_j$ is the deviation of the of the $j-$th sample.

Is there a way that I could compute these values in a "online" / "streaming way" (e.g., when the vectors are infinite) using only the aforementioned vectors?

Thanks in advance

Note that this question can be seem as complement to:

  • https://stats.stackexchange.com/questions/216047/how-does-one-go-about-determining-the-standard-deviation-of-an-entire-sample-dat, where the above formula for deviation is presented and
  • https://stats.stackexchange.com/questions/72212/updating-variance-of-a-dataset, where the mean and deviation are update at each new observation. In my case, I do not have access to the observations.
  • $\begingroup$ $1$. why do you have bars on your n's and s's? $2$. This question is already answered on site, for example, here: stats.stackexchange.com/questions/216047/… with pointers to other answers to the two subproblems (i.e. (i) calculating overall variance from subgroup means, variances or standard deviations and sample sizes and (ii) for online updating of variance) with some discussion of how they combine. $\endgroup$
    – Glen_b
    Commented Aug 8, 2019 at 3:47
  • $\begingroup$ If you need additional clarification, ask something specific relating to what you can't get from the existing answer (you can post a new question if it's not a simple clarification of the present one) $\endgroup$
    – Glen_b
    Commented Aug 8, 2019 at 3:57
  • $\begingroup$ Hi @Glen_b, thanks for the heads up. I have updated the question formulation. $\endgroup$ Commented Aug 8, 2019 at 4:28


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.