1
$\begingroup$

I'm having trouble finding a good resource on this.

I'm plotting some timeseries data over the last 200 years that has a clear trend, although there is also a lot of noise. I have smoothed the data using a simple centered moving average of all observations with +/- 2 years of each year (some years have many observations, some none).

I wanted to also plot a confidence interval of the moving average. Naively, I thought I could just find the standard error of the observations included in the window for each year. For example, my data might look like:

year value
4 1
4 2
5 2
7 5
7 4
8 7
9 5
10 9
10 8

For year 5, the observations in the window are 1,2,2,5,4. The moving average is 2.8. The standard deviation of these observations is 1.47, and the standard error is that divided by $\sqrt{5}$, which is 0.66. Assuming normal noise, this would give a 95% CI of +/- 1.96*0.66 = 1.29. That seems right to me but I cannot find a resource to confirm that.

Is that a reasonable thing to do? And could anyone point me to a resource that tells why that is/isn't ok

$\endgroup$
1
  • 1
    $\begingroup$ I think you have to know the theoretical distribution to do this. It was long impossible for exponential smoothing models to be assigned a confidence interval I believe until they were seen as a form of state space models when a distribution could be assigned. I would look up exponential smoothing/ confidence interval. $\endgroup$
    – user54285
    Commented May 17, 2021 at 21:49

1 Answer 1

3
$\begingroup$

While your approach is not wrong, it can be improved if you are willing to assume that the variance around the trend line is constant. In that case, it would be to estimate the standard error from all the residuals.

Let $y_t$ denote the observation at time $t$. Then the estimated trend is given by $$\hat{f}(t) = (y_{t-2} + y_{t-1} + y_t + y_{t+1} + y_{t+2})/5.$$ (Note that this doesn't need "centering" as it uses an odd number of observations.) The residuals are given by $e_t= y_t - f(t)$ and the estimated residual variance is $$\hat\sigma^2 = \frac{1}{T-4}\sum_{t=3}^{T-2} e_t^2,$$ where $T$ is the length of the series. The first two and last two trend estimates are missing due to insufficient observations at the start and end of the series.

A 95% confidence interval for the fitted line is $$\hat{f}(t) \pm 1.96 \frac{\hat{\sigma}}{\sqrt{5}}.$$

$\endgroup$
2
  • $\begingroup$ Thank you. I guess this would no longer apply in the case where I have irregular levels of observations along the trend line? For example in the 1820s I have hardly any observations, so my confidence interval in that range is currently higher than in the 1960s, where I have many more observations (similar residual variances). $\endgroup$
    – chasmani
    Commented May 18, 2021 at 7:47
  • $\begingroup$ You can't compute the trend at all if you don't have the neighbouring observations, so it is not just a problem in estimating the standard error. Try using a loess trend instead. $\endgroup$ Commented May 18, 2021 at 22:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.