# Forecasting average values with varying number of observations

Every day $t$ we observe several (independent) realizations of a variable $X_t$. This variable is the sum of a time dependent mean value and white noise:

$$X_t=\mu_t+\epsilon_t$$

You can assume $\epsilon_t$ has a fixed variance over time, is Gaussian and independent of everything. The underlying time series $\mu_t$ could be forecasted with usual time-series methods.

But $\mu_t$ is not directly observable. Instead we observe:

• $a_t$ the average observed value for $X_t$ over day $t$
• $n_t$ the number of observations for this day

The number $n_t$ can vary very much. The goal is not to forecast it but to take it into account to forecast $\mu_t$. Typically, when $n_t$ is small, we observe a noisy average. When $n_t$ is big, $a_t$ is very close to $\mu_t$ (the standard deviation of $a_t-\mu_t$ is $\frac{\sigma}{\sqrt n_t}$)

Do you know/think of a way to take $n_t$ into account when using forecasting methods? (Possibly very simple ones: assuming $\mu_t$ is just a random walk)

• Kalman filters might fit what you want, since they are intended to track a hidden variable with noisy measures. Your measures just get more noisy when sample size is small, but Kalman filters take that in account. Furthermore, since you know Et, you know how noisy is your measure, and that should ease your task.
– Pere
Commented Nov 13, 2017 at 15:45
• Many thanks. It was what I was looking for. I posted my answer after understand all this. Commented Nov 16, 2017 at 14:02

The idea is to use the Kalman filter. An easy to read (but quite complete) introduction is available here: How a Kalman filter works, in pictures. It is essentially Gaussian Bayesian inference.

Assume $\mu_t$ is just a random walk with Gaussian step of known variance $1$. I explain how to implement the Kalman filter in this simple case.

At each time $t$ the filter saves its knowledge about $\mu_t$ as a Gaussian distribution for $\mu_t$ with mean $m$ and variance $s^2$. When you move forward into the future, you loose information because $\mu_t$ varies randomly with variance $1$. The filter is updated as such:

$$m\leftarrow m \\ s^2\leftarrow s^2+1\\$$

When you observe $x$ a realization of $X_t$, the filter does a Bayesian inference resulting in: $$m\leftarrow \frac{\frac{1}{\sigma^2}x+\frac{1}{s^2}m}{\frac{1}{\sigma^2}+\frac{1}{s^2}} \\ s^2\leftarrow \frac{1}{\frac{1}{\sigma^2}+\frac{1}{s^2}}\\$$

You can manage $n$ observations as a single update replacing $x$ by the average value $a_t$ and $\sigma^2$ by $\sigma^2/n_t$.

In other words, when you move forward into the future, the uncertainty $s^2$ increases. After each observation the uncertainty $s^2$ decreases and the forecast $m$ is updated with weights depending on the present uncertainty and the number of observations.

Red: observations. Blue: $m$ after observations

Interesting question ! If you model the average as a function of N and find that the variance of the conditional average is related to N then this might suggest a Box-Cox value of 0 . Why don't you post data for 100 consecutive days and I will try to investigate your assertion.