Forecasting average values with varying number of observations

Question

Every day $t$ we observe several (independent) realizations of a variable $X_t$. This variable is the sum of a time dependent mean value and white noise:

$$X_t=\mu_t+\epsilon_t$$

You can assume $\epsilon_t$ has a fixed variance over time, is Gaussian and independent of everything. The underlying time series $\mu_t$ could be forecasted with usual time-series methods.

But $\mu_t$ is not directly observable. Instead we observe:

• $a_t$ the average observed value for $X_t$ over day $t$
• $n_t$ the number of observations for this day

The number $n_t$ can vary very much. The goal is not to forecast it but to take it into account to forecast $\mu_t$. Typically, when $n_t$ is small, we observe a noisy average. When $n_t$ is big, $a_t$ is very close to $\mu_t$ (the standard deviation of $a_t-\mu_t$ is $\frac{\sigma}{\sqrt n_t}$)

Do you know/think of a way to take $n_t$ into account when using forecasting methods? (Possibly very simple ones: assuming $\mu_t$ is just a random walk)

Answer 1

The idea is to use the Kalman filter. An easy to read (but quite complete) introduction is available here: How a Kalman filter works, in pictures. It is essentially Gaussian Bayesian inference.

Assume $\mu_t$ is just a random walk with Gaussian step of known variance $1$. I explain how to implement the Kalman filter in this simple case.

At each time $t$ the filter saves its knowledge about $\mu_t$ as a Gaussian distribution for $\mu_t$ with mean $m$ and variance $s^2$. When you move forward into the future, you loose information because $\mu_t$ varies randomly with variance $1$. The filter is updated as such:

$$m\leftarrow m \\ s^2\leftarrow s^2+1\\$$

When you observe $x$ a realization of $X_t$, the filter does a Bayesian inference resulting in: $$m\leftarrow \frac{\frac{1}{\sigma^2}x+\frac{1}{s^2}m}{\frac{1}{\sigma^2}+\frac{1}{s^2}} \\ s^2\leftarrow \frac{1}{\frac{1}{\sigma^2}+\frac{1}{s^2}}\\$$

You can manage $n$ observations as a single update replacing $x$ by the average value $a_t$ and $\sigma^2$ by $\sigma^2/n_t$.

In other words, when you move forward into the future, the uncertainty $s^2$ increases. After each observation the uncertainty $s^2$ decreases and the forecast $m$ is updated with weights depending on the present uncertainty and the number of observations.

Red: observations. Blue: $m$ after observations

Answer 2

Interesting question ! If you model the average as a function of N and find that the variance of the conditional average is related to N then this might suggest a Box-Cox value of 0 . Why don't you post data for 100 consecutive days and I will try to investigate your assertion.