# Estimate in presence of missing observations

I'm trying to estimate a parameter based on its past history. However, I do not have the observed data at every point of time. To illustrate the scenario, consider a group of N people where each person estimates how many email the remaining N-1 persons would receive the next day. These people meet infrequently; when X meets Y, they only exchange the count of emails received yesterday i.e. no historic data is available.

Would it be right to use exponential weighted moving average (EWMA) in this case?

$$\hat{X}_t = a * X_t + (1 - a) * \hat{X}_{t-1}$$

To be precise: I'm trying to estimate (or rather, forecast) a value based on historical values where some observations could be missing. I could use either EWMA of Kalman filter (or anything else) -- point is which method would be robust in case of such missing observations?

-

Sounds like a state-space model estimable with Kalman filter to me (although in your example, obviously made up to cover the secret details of your actual application, the data are discrete).

You have an unobservable state $X_t$ that evolves according to an equation like $$x_t = A x_{t-1} + \epsilon_t$$ What you observe (at least for some periods of time) is an imperfect measurement of it, $Y_t$, given by $$y_t = Bx_t + \delta_t$$ In your application, $x_t$ may be univariate reflecting an overall Internet activity rate, with $A$ being the vector of multipliers of the individual activity/connectedness, or multivariate with entries reflecting the email rate for each of the participants. $y_t$ would be the infrequently observed actual count, so $A$ would be a unit matrix.

By assuming multivariate normality of $x_t$, $\epsilon_t$, $\delta_t$, as well as some identifying constraints on the covariance matrices of these variables (or vectors -- e.g., independence of $\delta_t$ and $\epsilon_t$ from each other and from the history $(x_s,y_s), s=t-1, t-2, \ldots$), you can manipulate the conditional distributions and obtain improved predictions $x_t|\{x_s, y_s, y_t\}$ of the state vector $x_t$. The formulae are provided on the Wikipedia page I linked.

-
This sounds promising to me! The example given is almost similar to my actual problem with other added complexities. Is there any tutorial available on Kalman filters for absolute beginners? Perhaps, with some worked out examples to illustrate how it could be used in real life? –  Barun Apr 24 '12 at 9:31
Here is a very simple introduction to the Kalman filter, to estimate the position of a robot (think of the position as the parameter you are trying to estimate): sites.google.com/site/udacitymirrorcs373/cs-373/unit-2 (you may want to skip part of the begining, which is irrelevant, and check the previous and next lectures, which present non parametric alternatives to the Kalman filter: histogram filter and particle filter). –  Vincent Zoonekynd Apr 25 '12 at 3:07
@VincentZoonekynd: Thanks! Looks interesting. I would go through this. –  Barun May 10 '12 at 11:14