Calculate average of a set numbers with reported standard errors

Question

I have 365 daily measurements that all have standard errors associated with them.

    Date        | Prediction | Standard Error
    -----------------------------------------   
    Jan-01-2003 | 24.8574    | 10.6407
    Jan-02-2003 | 10.8658    | 3.8237
    Jan-03-2003 | 12.1917    | 5.7988
    Jan-04-2003 | 11.1783    | 4.3016
    Jan-05-2003 | 16.713     | 5.3177
    etc ...

What is the statistically appropriate way of getting the yearly average with a 95% Confidence Interval around it ? I am assuming that the errors must be propagating somehow and need to be accounted for.

Google returns mostly information on how to calculate the average or standard deviation of a set of numbers, not a set of numbers with errors.

I would also appreciate some type of internet reference so I can refer to it later.

Answer 1

"The variance of the sums is the sum of the variances". So:

Square each of the 365 standard errors so they become variances. Add them together; this will give you the variance of the annual total. Divide that variance by 365^2; this will give you the variance of the annual average. Take the square root of that variance; this will give you the standard error of your annual average.

From there, I suspect your sample size is big enough (bigger than 500 in total, right?) it doesn't matter too much what the underlying population is (log normal etc) as your estimate is probably roughly normally distributed due to the central limit theorem. So multiply the standard error calculated above by 1.96 to give the +/- of your 95 percent confidence interval.

Edit / addition

On reflection, my answer above is probably incomplete. I should have asked you for more context. Most importantly there is a question about where your original data come from. Are they themselves the averages of sets of independent observations? (or something similar, eg output from a regression) If so, are they each based on the same number of observations? If not ie if they are based on different numbers of observations, you will need a weighted average, and hence a weighted estimate of the variance. Arguably, you may want to do this anyway. In this case, weighting should be proportional to the inverse of the variance of each "prediction" in your data set; or to the number of observations behind each "prediction".

Answer 2

In Ken Tatebe’s “Combining Multiple Averaged Data Points And Their Errors” (PDF—linked in Steven Howell’s answer), Tatebe shows that given two "bins" of averaged data points $a$ and $b$, each reported as an average ($\bar a$, $\bar b$) and an error ($\varepsilon_a$, $\varepsilon_b$), with bin sizes $n_a$ and $n_b$ (where $n_a$ + $n_b$ = $n$ is the number of original data points), the average of the entire data set $\bar S$ is given by the weighted average of the averages:

$$ \bar S = (\frac{n_a}{n}) \bar a + (\frac{n_b}{n}) \bar b $$

and the error $\varepsilon_S$ is given by

$$ \varepsilon_S = \sqrt{\frac{N_a}{N}\varepsilon^2_a+\frac{N_b}{N}\varepsilon^2_b+\frac{n_an_b(\bar a - \bar b)^2}{nN}} $$

Tatebe goes on to state:

If more than two averaged points need to be combined, the formulae may be used repeatedly to combine multiple sets. For example, if one has three averaged data points $\bar a$, $\bar b$ and $\bar c$ one may use Equations 10 and 48…

(equations 10 and 48 being $\bar S$ and $\varepsilon_S$ above, respectively)

…to combine $\bar a$ and $\bar b$. This result can then be combined with $\bar c$ to get the final composite point.

I’ve written a Python implementation; it’s untested, but in case it’s useful to anyone else, it’s available here as a gist.