100
$\begingroup$

I have a monthly average for a value and a standard deviation corresponding to that average. I am now computing the annual average as the sum of monthly averages, how can I represent the standard deviation for the summed average ?

For example considering output from a wind farm:

Month        MWh     StdDev
January      927     333 
February     1234    250
March        1032    301
April        876     204
May          865     165
June         750     263
July         780     280
August       690     98
September    730     76
October      821     240
November     803     178
December     850     250

We can say that in the average year the wind farm produces 10,358 MWh, but what is the standard deviation corresponding to this figure ?

$\endgroup$
11
  • 7
    $\begingroup$ A discussion following a now-deleted reply noted a possible ambiguity in this question: do you seek the SD of the monthly averages or do you want to recover the SD of all the original values from which those averages were constructed? That reply also correctly pointed out that if you want the latter, you will need the numbers of values involved in each one of the monthly averages. $\endgroup$
    – whuber
    Apr 4, 2012 at 17:37
  • 2
    $\begingroup$ A comment to another deleted reply pointed out that it is strange to compute an average as a sum: surely you mean that you are averaging the monthly averages. But if what you want is to estimate the average of all the original data, then such a procedure is not usually a good one: a weighted average is needed. And of course it's not possible to give a good answer to your question about the "SD for the summed average" until it is clear what the "summed average" is and what it is intended to represent. Please clarify that for us. $\endgroup$
    – whuber
    Apr 4, 2012 at 21:40
  • 1
    $\begingroup$ @whuber I have added an example to clarify. Mathematically I believe that the sum of averages is equal to the monthly average times 12. $\endgroup$
    – klonq
    Apr 5, 2012 at 6:37
  • 3
    $\begingroup$ Yes, klonq, that is a very reasonable request. However, these replies were deleted by their owner, not by the community. To preserve their value, I have attempted here to relay (my take on) the key ideas arising in those replies and their comments. BTW, your recent edits are quite helpful: people like to see example data. $\endgroup$
    – whuber
    Apr 5, 2012 at 14:33
  • 1
    $\begingroup$ Surely averaging the variance and thus calculating the average standard deviation can't be the whole answer! All this represents is the average variance in measuring power output WITHIN a single month. This is a good start at getting an accurate guage on measurement error but doesn't this standard deviation of 232 need to be combined in some way with the INTER-MONTHLY variation in power output. i.e. I think that the end resulting standard deviation for the Grand Mean should be a little higher than 232 if you account for the combined error in measurement of both within each month as well as BET $\endgroup$
    – user46421
    May 30, 2014 at 3:49

6 Answers 6

90
$\begingroup$

Short Answer

You average the variances; then you can take square root to get the average standard deviation.

For example:

╔═══════════╦════════╤════════╤══════════╗
║ Month     ║    MWh │ StdDev │ Variance ║
╠═══════════╬════════╪════════╪══════════╣
║ January   ║    927 │    333 │  110,889 ║
║ February  ║  1,234 │    250 │   62,500 ║
║ March     ║  1,032 │    301 │   90,601 ║
║ April     ║    876 │    204 │   41,616 ║
║ May       ║    865 │    165 │   27,225 ║
║ June      ║    750 │    263 │   69,169 ║
║ July      ║    780 │    280 │   78,400 ║
║ August    ║    690 │     98 │    9,604 ║
║ September ║    730 │     76 │    5,776 ║
║ October   ║    821 │    240 │   57,600 ║
║ November  ║    803 │    178 │   31,684 ║
║ December  ║    850 │    250 │   62,500 ║
╠═══════════╬════════╪════════╪══════════╣
║ Total     ║ 10,358 │    805 │  647,564 ║
║ ÷12       ║    863 │    232 │   53,964 ║
╚═══════════╩════════╧════════╧══════════╝

And then the average standard deviation is sqrt(53,964) = 232

Long Answer

From Sum of normally distributed random variables:

If $X$ and $Y$ are independent random variables that are normally distributed (and therefore also jointly so), then their sum is also normally distributed

...the sum of two independent normally distributed random variables is normal, with its mean being the sum of the two means, and its variance being the sum of the two variances

And from Wolfram Alpha's Normal Sum Distribution:

Amazingly, the distribution of a sum of two normally distributed independent variates $X$ and $Y$ with means and variances $(\mu_X,\sigma_X^2)$ and $(\mu_Y,\sigma_Y^2)$, respectively is another normal distribution

$$ P_{X+Y}(u) = \frac{1}{\sqrt{2\pi (\sigma_X^2 + \sigma_Y^2)}} e^{-[u-(\mu_X+\mu_Y)]^2/[2(\sigma_X^2 + \sigma_Y^2)]} $$

which has mean

$$\mu_{X+Y} = \mu_X+\mu_Y$$

and variance

$$ \sigma_{X+Y}^2 = \sigma_X^2 + \sigma_Y^2$$

For your data:

  • sum: 10,358 MWh
  • variance: 647,564
  • standard deviation: 804.71 (i.e. sqrt(647,564))

enter image description here

So to answer your question:

  • How to 'sum' a standard deviation?

  • You sum them quadratically:

      s = sqrt(s1^2 + s2^2 + ... + s12^2)
    

Conceptually you sum the variances, then take the square root to get the standard deviation.


Because i was curious, i wanted to know the average monthly mean power, and its standard deviation. Through induction, we need 12 normal distributions which:

  • sum to a mean of 10,358
  • sum to a variance of 647,564

That would be 12 average monthly distributions of:

  • mean of 10,358/12 = 863.16
  • variance of 647,564/12 = 53,963.6
  • standard deviation of sqrt(53963.6) = 232.3

enter image description here

We can check our monthly average distributions by adding them up 12 times, to see that they equal the yearly distribution:

  • Mean: 863.16*12 = 10358 = 10,358 (correct)
  • Variance: 53963.6*12 = 647564 = 647,564 (correct)

Note: i'll leave it to someone with a knowledge of the esoteric Latex math to convert my formula images, and formula code into stackexchange formatted formulas.

Edit: I moved the short, to the point, answer up top. Because i needed to do this again today, but wanted to double-check that i average the variances.

$\endgroup$
13
  • 4
    $\begingroup$ This all seems to assume the months are uncorrelated - have you made that assumption explicit anywhere? Also, why do we need to bring in the normal distribution? If we're only talking about variance then that seems unnecessary - for example, see my answer here $\endgroup$
    – Macro
    Jul 25, 2012 at 12:26
  • 1
    $\begingroup$ @Marco Because i think better in pictures and it makes everything easier to understand. $\endgroup$
    – Ian Boyd
    Jul 25, 2012 at 19:57
  • 2
    $\begingroup$ @Marco Also, i believe this question started on the (now defunct) stats.stackexchange site. A wall of formulas are less accessible than simpler, graphical, less rigorous treatments. $\endgroup$
    – Ian Boyd
    Jul 26, 2012 at 13:45
  • 3
    $\begingroup$ I doubt this is correct. Imagine two data sets with each only a single measurement each. Their variance of each set is 0, but the set of both measurements has a variance greater than 0 if the data points differ. $\endgroup$
    – Njol
    Oct 10, 2016 at 15:58
  • 1
    $\begingroup$ @Njol, I think that's why we assume all variables have normal distribution. And we can do it here, because we talk about phisical measurement. In your example both variables are not normally distributed. $\endgroup$
    – tworec
    Apr 26, 2017 at 12:56
27
$\begingroup$

This is an old question but the answer accepted is incorrect or at least incomplete. The user wants to calculate the standard deviation over 12-month data where the mean and standard deviation are already calculated over each month. Assuming that the number of samples in each month is the same, then it is possible to calculate the sample mean and variance over the year from each month's data. For simplicity assume that we have two sets of data:

$X=\{x_1,....x_N\}$

$Y=\{y_1,....,y_N\}$

with known values of sample mean and sample variance, $\mu_x$, $\mu_y$,$\sigma^2_x$,$\sigma^2_y$.

Now we want to calculate the same estimates for

$Z=\{x_1,....,x_N, y_1,...,y_N\}$.

Consider that $\mu_x$,$\sigma^2_x$ are calculated as:

$\mu_x = \frac{\sum^N_{i=1} x_i}{N}$

$\sigma^2_x = \frac{\sum^N_{i=1} x^2_i}{N}-\mu^2_x$

To estimate mean and variance over the total set we need to calculate:

$\mu_z = \frac{\sum^N_{i=1} x_i +\sum^N_{i=1} y_i }{2N}= (\mu_x+\mu_y)/2$ which is given in the accepted answer. For variance however the story is different:

$\sigma^2_z = \frac{\sum^N_{i=1} x^2_i +\sum^N_{i=1} y^2_i }{2N}-\mu^2_z$

$\sigma^2_z = \frac{1 }{2}(\frac{\sum^N_{i=1} x^2_i}{N}-\mu^2_x + \frac{\sum^N_{i=1} y^2_i}{N}-\mu^2_y )+\frac{1 }{2}(\mu^2_x+\mu^2_y) -(\frac{\mu_x+\mu_y}{2})^2$

$\sigma^2_z = \frac{1 }{2}(\sigma^2_x+\sigma^2_y )+(\frac{\mu_x-\mu_y}{2})^2$

So if you have the variance over each subset and you want the variance over the whole set then you can average the variances of each subset if they all have the same mean. Otherwise, you need to add the variance of the mean of each subset.

As an example assume that over the first half of the year we produce exactly 1000 MWh per day and in the second half, we produce 2000 MWh per day. Then the mean and variance of energy production in the first and the second half are 1000 and 2000 for the mean and 0 for the variance of each half year. Now we want to calculate the variance of energy production over the whole year. If we average the two variances we arrive at zero, which is not correct since the energy per day over the whole year is not constant. Hence we need to add the variance of all the means from each subset.

This has a close connection to the law of total variance. enter link description here.

$$\operatorname{Var}(Y) = \operatorname{E}[\operatorname{Var}(Y \mid X)] + \operatorname{Var}(\operatorname{E}[Y \mid X])$$

To use the above theorem in this case, we can interpret the conditioning variable X as Y belongs to group $X_i$. In the context of the original question, X is the random variable indicating the month of the year and Y is energy production per day.

$\endgroup$
5
  • 1
    $\begingroup$ Nice answer. In my opinion, how to calculate it depends on how you want to present the resultant SD (and what hypothesis you want to address using this SD, if you are trying to compare to another windfarm etc.). $\endgroup$
    – y chung
    Mar 24, 2020 at 3:41
  • $\begingroup$ You meant "population variance" not "sample variance" right? Sample variance has that annoying N-1 in its definition. $\endgroup$
    – ions me
    Nov 26, 2022 at 22:02
  • $\begingroup$ Nice answer! In case the number of samples in each subset is not same and their averages also are not same, then would the formula look like below?: $\frac{(n_1-1) \sigma_1^2 + ... + (n_k-1) \sigma_k^2}{n_1+...+n_k -k} + (\frac{\mu_1-...-\mu_k}{n_1+...+n_k})^2$ $\endgroup$
    – Chaos
    Jun 12, 2023 at 8:22
  • 1
    $\begingroup$ In the first part the dominator should be $ \sum_i N_i-1$ instead of $\sum_i N_i-k$. It should not be affected by the number of groups. otherwise, the variance can be changed by making more groups. The second part should be $\frac{1}{N-1}\sum_i N_i (\mu_i - \bar\mu)^2$ where $\bar\mu =\frac{1}{N} \sum_i N_i \mu_i $ and $N = \sum_i N_i$ $\endgroup$
    – Hooman
    Jun 13, 2023 at 9:14
  • $\begingroup$ How do you compute the law of total variance when using Xi instead of X? We have a partition of X into {X1, X2, ..., Xk} $\endgroup$ Dec 28, 2023 at 23:58
11
$\begingroup$

TL;DR

Given several days, and for each day we are given its Average, Sample StdDev and number of Samples, denoted as: $$ \mu_d,\ \sigma_d,\ N_d $$ We would like to compute the Average and Sample StdDev across all days.

Average is simply a weighted average: $$ \mu = \frac{\sum{\mu_dN_d}}{\sum{N_d}} = \frac{\sum{\mu_dN_d}}{N} $$

Sample StdDev is this thing: $$ \sigma=\sqrt{\frac{\sum_{d}{(\sigma_d^2(N_d-1)+N_d(\mu-\mu_d)^2})}{N-1}} $$ Where subscript d denotes a day we collected Average, Sample StdDev and number of Samples for.

Details

We've had a similar problem in which we had a process that computes a daily Average and Sample StdDev and saves it alongside the number of daily samples. Using this input we had to compute a weekly / monthly Average and StdDev. The number of samples per day was not constant in our case.

Denote the Average, Sample StdDev and Number of Samples of the entire set as: $$ \mu,\ \sigma\ and\ N\ $$ And for day d denote the Average, Sample StdDev and Number of Samples as: $$ \mu_d,\ \sigma_d,\ N_d $$ Computing the entire set's Average is simply a a Weighted Average of the days' Averages in question: $$ \mu = \frac{\sum{\mu_dN_d}}{\sum{N_d}} = \frac{\sum{\mu_dN_d}}{N} $$ But things are much more involved when considering Sample StdDev. For a day's Sample StdDev we have: $$ \sigma_d=\sqrt{\frac{\sum_{N_d}(x_j-\mu_d)^2}{N_d-1}} $$ First a bit of clean up: $$ \sigma_d^2(N_d-1)=\sum_{N_d}(x_j-\mu_d)^2 $$ Let's look at the right-hand side term of the equation above. If we can reach from this sum to the following sum per day: $$ \sum_{N_d}{(x_j-\mu)^2} $$ then summation over the days will give us what we are looking for as the days are disjoint and cover the entire set: $$ \sum_{d}{\sum_{N_d}{(x_j-\mu)^2}} = \sum_{N}{(x_j-\mu)^2} $$ The insight to get from daily StdDev to the entire set's StdDev is to notice that while we don't have the daily samples, we do have the sum of the daily samples through the daily Average. Given this insight let's work on the right-hand side term of the equation above: $$ \sum_{N_d}(x_j-\mu_d)^2=\sum_{N_d}{(x_j^2-2x_j\mu_d+\mu_d^2)}=\\ =\sum_{N_d}{(x_j^2-2x_j\mu_d+\mu_d^2)}+(\sum_{N_d}{\mu^2}-\sum_{N_d}{\mu^2})+(2\sum_{N_d}{x_j(\mu-\mu_d})-2\sum_{N_d}{x_j(\mu-\mu_d})) $$ At this point we did nothing but adding and subtracting terms that will zero out keeping the equation the same. Now since we sum over Nd on all summations let's rewrite the summations for fun and profit: $$ \require{cancel} =\sum_{N_d}{(x_j^2-2x_j(\cancel{\mu_d}+\mu-\cancel{\mu_d})+\mu^2)}+\sum_{N_d}{\mu_d^2}-\sum_{N_d}{\mu^2}+2\sum_{N_d}{x_j(\mu-\mu_d}) $$ Summations are over j so summation terms that are not dependent on j can be simply multiplied by Nd: $$ =\sum_{N_d}{(x_j^2-2x_j\mu+\mu^2)}+N_d\mu_d^2-N_d\mu^2+2\sum_{N_d}{x_j(\mu-\mu_d)} $$ And we are getting close: $$ =\sum_{N_d}{(x_j-\mu)^2}+N_d\mu_d^2-N_d\mu^2+2\sum_{N_d}{x_j(\mu-\mu_d)} $$ Now let's handle the rightmost term as we can't use xj directly but we can use its sum as we have that day's Average. Simply multiply and divide by Nd to get the Average: $$ =\sum_{N_d}{(x_j-\mu)^2}+N_d\mu_d^2-N_d\mu^2+2(\mu-\mu_d){N_d}(\frac{1}{N_d}\sum_{N_d}{x_j})\\ =\sum_{N_d}{(x_j-\mu)^2}+N_d\mu_d^2-N_d\mu^2+2(\mu-\mu_d){N_d}\mu_d $$ At this point we have the summation we need to compute the entire set's Sample StdDev and all the other terms are quantities we know, namely day's statistics and number of samples. Let's plug it back to the clean-up step above: $$ \sigma_d^2(N_d-1)=\sum_{N_d}{(x_j-\mu)^2}+N_d\mu_d^2-N_d\mu^2+2(\mu-\mu_d){N_d}\mu_d\\ \leftrightarrow\ \sigma_d^2(N_d-1)-N_d\mu_d^2+N_d\mu^2-2N_d\mu_d(\mu-\mu_d)=\sum_{N_d}{(x_j-\mu)^2}\\ \leftrightarrow\ \sigma_d^2(N_d-1)+N_d(\mu-\mu_d)^2=\sum_{N_d}{(x_j-\mu)^2} $$ We are now ready to compute the set's Sample StdDev: $$ \sigma=\sqrt{\frac{\sum_{N}(x_j-\mu)^2}{N-1}}\\ =\sqrt{\frac{\sum_{d}{\sum_{N_d}(x_j-\mu)^2}}{N-1}}\\ =\sqrt{\frac{\sum_{d}{(\sigma_d^2(N_d-1)+N_d(\mu-\mu_d)^2})}{N-1}} $$

$\endgroup$
6
  • $\begingroup$ Your notation is a bit confusing to me as it doesn't make clear which means & standard deviations are known (assumed) parameters & which are sample estimates. $\endgroup$ Dec 23, 2019 at 21:46
  • 2
    $\begingroup$ Knowns are Nd, Mu-d,Sigma-d, we need to compute N, Mu, Sigma. Computing N and Mu is trivial, Sigma is the involved one.. $\endgroup$
    – YoavG
    Dec 23, 2019 at 21:50
  • $\begingroup$ Thanks. The key observation is that the number of samples differ from stda to stdb. Most of the other answers ignores that... $\endgroup$
    – vpuente
    Oct 27, 2022 at 14:35
  • 1
    $\begingroup$ Note that $N$ is the total number of data points, not the number of days. So $N = \sum_d N_d$. This tripped me up at first. $\endgroup$ May 7, 2023 at 19:02
  • $\begingroup$ Assuming that there were $m$ days ; shouldn't the denominator of the final formula for $\sigma$ be $N-m$ instead of $N-1$? $\endgroup$
    – Chaos
    Jun 12, 2023 at 9:28
8
$\begingroup$

I'd like to stress again the incorrectness in part of the accepted answer. The wording of the question lead to confusion.

The question have Average and StdDev of each month, but it's unclear what kind of subset is used. Is it the average of 1 wind turbine of the whole farm or the daily average of the whole farm? If it's the daily average for each month, you can't add up the monthly average to get the annual average because they do not have the same denominator. If it's the unit average, the question should state

We can say that in the average year each turbine in the wind farm produces 10,358 MWh,...

Instead of

We can say that in the average year the wind farm produces 10,358 MWh,...

Further more, The Standard deviation or variance is the comparison against the set's own average. It does NOT contain any information regarding the average of its parent set (the bigger set which the computed set is a component of).

Short answer: You average the variances; then you can take square root to get the average standard deviation.

This is wrong. You can do normal math operation with averages (without involving the elements' data) because they share common denominators. NOT the case with variance nor standard deviation

Variance visualization

The image is not necessarily very precise, but it conveys the general idea. Let's imagine the output of one wind farm as in the image. As you can see, the "local" variance has nothing to do with the "global" variance, no matter how you add or multiply those. If you add the "local" variances together, it will be very small compare to the "global" variance. You cannot predict the variance of the year using variance of 2 half year. So, in the accepted answer, while the sum calculation is correct, the division by 12 to get the monthly number means nothing.. Of the three sections, the first and last sections are wrong, the second is right.

Again, it's a very wrong application, please do not follow it or it will get you into trouble. Just calculate for the whole thing, using total yearly/monthly output of each unit as data points depending whether you want yearly or monthly number, that should be the correct answer. You probably want something like this. This is my randomly generated numbers. If you have the data, the result in cell O2 should be your answer.

enter image description here

$\endgroup$
2
  • $\begingroup$ Thank you very much for the image which helped me a lot to understand why the accepted answer is incomplete and may be even wrong. You explained it very well, thank you! $\endgroup$
    – Kay
    Sep 9, 2019 at 10:25
  • 2
    $\begingroup$ This show the danger of voting. The people who vote are the people who doesn't know the answer. As oppose to coding, the people who vote is people who get the code working, the more vote, the better the answer. For statistic/math, more votes only means it's more appealing. $\endgroup$
    – Tam Le
    Sep 11, 2019 at 4:33
1
$\begingroup$

I believe what you may be really interested in though is the standard error rather than the standard deviation.

The standard error of the mean (SEM) is the standard deviation of the sample-mean's estimate of a population mean, and that will give you a measure how how good your yearly MWh estimate is.

It's very easy to compute: if you used $n$ samples to obtain your monthly MWh averages and standard deviations, you would just compute the standard deviation as @IanBoyd suggested and normalize it by the total size of your sample. That is,
$$ s = \frac{\sqrt{s_1^2 + s_2^2 + \ldots + s_{12}^2}}{\sqrt{12 \times n}} $$

$\endgroup$
1
$\begingroup$

If you know the number of samples used for the calculation of the monthly mean and standard deviation, you can use the "batch extension" by Chan et al. of Welford's algorithm to combine the variances (squares of standard deviations) and means of data subsets. The algorithm is numerically robust and exact.

See this Wiki page.

I have implemented it in Python here.

For your example, and additionally an assumed sample size of 30 per month, the usage and result would be

mean_array = [927,1234,1032,876,865,750,780,690,730,821,803,850]
stddev_array =[333 ,250,301,204,165,263,280,98,76,240,178,250]

# number of samples for the monthly mean and standard dev
n_array =[30,30,30,30,30,30,30,30,30,30,30,30]

sa = StatisticsAggregator()
for n,mean,stddev in zip(n_array, mean_array, stddev_array):
    sa.add(n, mean, stddev)

print('global mean', sa.mean)
print('global std. dev.', np.sqrt(sa.var))

#>> global mean 863.1666666666666
#>> global std. dev. 143.15424858832827
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.