10
$\begingroup$

I have 365 daily measurements that all have standard errors associated with them.

    Date        | Prediction | Standard Error
    -----------------------------------------   
    Jan-01-2003 | 24.8574    | 10.6407
    Jan-02-2003 | 10.8658    | 3.8237
    Jan-03-2003 | 12.1917    | 5.7988
    Jan-04-2003 | 11.1783    | 4.3016
    Jan-05-2003 | 16.713     | 5.3177
    etc ...

What is the statistically appropriate way of getting the yearly average with a 95% Confidence Interval around it ? I am assuming that the errors must be propagating somehow and need to be accounted for.

Google returns mostly information on how to calculate the average or standard deviation of a set of numbers, not a set of numbers with errors.

I would also appreciate some type of internet reference so I can refer to it later.

$\endgroup$
9
  • $\begingroup$ Do you know if the data normally distributed? $\endgroup$
    – ahoffer
    Commented Jan 13, 2012 at 22:06
  • $\begingroup$ I do not. For sake of argument we can say it is but it is likely Poisson because much of the other data I work with usually is. $\endgroup$
    – user918967
    Commented Jan 14, 2012 at 5:15
  • $\begingroup$ The Poisson distribution is used for discrete data whereas your data seems to be continuous. What I would like to know is how the standard errors were obtained. Are they related to the measrements themselves or were they somehow obtained separately? $\endgroup$
    – MånsT
    Commented Jan 15, 2012 at 9:11
  • $\begingroup$ An average is just a the sum of each item times its proportion. In the case of a normal average these would just be equal for each item (summing to 1 of course). So Is it appropriate to just use normal addition error propagation after multiplying by the proportion? $\endgroup$
    – DQdlM
    Commented Jan 15, 2012 at 17:18
  • 1
    $\begingroup$ The way the standard error of the sum (and hence the average) works depends on the assumptions made and how the predictions are generated (which will impact the correlation between them) $\endgroup$
    – Glen_b
    Commented Sep 25, 2015 at 9:39

2 Answers 2

7
$\begingroup$

"The variance of the sums is the sum of the variances". So:

Square each of the 365 standard errors so they become variances. Add them together; this will give you the variance of the annual total. Divide that variance by 365^2; this will give you the variance of the annual average. Take the square root of that variance; this will give you the standard error of your annual average.

From there, I suspect your sample size is big enough (bigger than 500 in total, right?) it doesn't matter too much what the underlying population is (log normal etc) as your estimate is probably roughly normally distributed due to the central limit theorem. So multiply the standard error calculated above by 1.96 to give the +/- of your 95 percent confidence interval.

Edit / addition

On reflection, my answer above is probably incomplete. I should have asked you for more context. Most importantly there is a question about where your original data come from. Are they themselves the averages of sets of independent observations? (or something similar, eg output from a regression) If so, are they each based on the same number of observations? If not ie if they are based on different numbers of observations, you will need a weighted average, and hence a weighted estimate of the variance. Arguably, you may want to do this anyway. In this case, weighting should be proportional to the inverse of the variance of each "prediction" in your data set; or to the number of observations behind each "prediction".

$\endgroup$
3
  • $\begingroup$ "Divide that variance by 365; this will give you the variance of the annual average." Shouldn't that be 365^2? $\endgroup$
    – zbicyclist
    Commented Jan 17, 2012 at 2:25
  • $\begingroup$ Yes, well spotted, sorry; have corrected. $\endgroup$ Commented Jan 17, 2012 at 3:38
  • $\begingroup$ The edit is getting closer. This appears to be a components of variance problem: we should be estimating the variance of the "predictions" and then using that together with the individual variances to weight the mean to obtain an approximate MVUE. $\endgroup$
    – whuber
    Commented Jan 18, 2012 at 23:40
2
$\begingroup$

In Ken Tatebe’s “Combining Multiple Averaged Data Points And Their Errors” (PDF—linked in Steven Howell’s answer), Tatebe shows that given two "bins" of averaged data points $a$ and $b$, each reported as an average ($\bar a$, $\bar b$) and an error ($\varepsilon_a$, $\varepsilon_b$), with bin sizes $n_a$ and $n_b$ (where $n_a$ + $n_b$ = $n$ is the number of original data points), the average of the entire data set $\bar S$ is given by the weighted average of the averages:

$$ \bar S = (\frac{n_a}{n}) \bar a + (\frac{n_b}{n}) \bar b $$

and the error $\varepsilon_S$ is given by

$$ \varepsilon_S = \sqrt{\frac{N_a}{N}\varepsilon^2_a+\frac{N_b}{N}\varepsilon^2_b+\frac{n_an_b(\bar a - \bar b)^2}{nN}} $$

Tatebe goes on to state:

If more than two averaged points need to be combined, the formulae may be used repeatedly to combine multiple sets. For example, if one has three averaged data points $\bar a$, $\bar b$ and $\bar c$ one may use Equations 10 and 48…

(equations 10 and 48 being $\bar S$ and $\varepsilon_S$ above, respectively)

…to combine $\bar a$ and $\bar b$. This result can then be combined with $\bar c$ to get the final composite point.

I’ve written a Python implementation; it’s untested, but in case it’s useful to anyone else, it’s available here as a gist.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.