Estimate in presence of missing observations

Question

I'm trying to estimate a parameter based on its past history. However, I do not have the observed data at every point of time. To illustrate the scenario, consider a group of N people where each person estimates how many email the remaining N-1 persons would receive the next day. These people meet infrequently; when X meets Y, they only exchange the count of emails received yesterday i.e. no historic data is available.

Would it be right to use exponential weighted moving average (EWMA) in this case?

$$ \hat{X}_t = a * X_t + (1 - a) * \hat{X}_{t-1} $$

To be precise: I'm trying to estimate (or rather, forecast) a value based on historical values where some observations could be missing. I could use either EWMA of Kalman filter (or anything else) -- point is which method would be robust in case of such missing observations?

Answer 1

Sounds like a state-space model estimable with Kalman filter to me (although in your example, obviously made up to cover the secret details of your actual application, the data are discrete).

You have an unobservable state $X_t$ that evolves according to an equation like $$x_t = A x_{t-1} + \epsilon_t$$ What you observe (at least for some periods of time) is an imperfect measurement of it, $Y_t$, given by $$y_t = Bx_t + \delta_t$$ In your application, $x_t$ may be univariate reflecting an overall Internet activity rate, with $A$ being the vector of multipliers of the individual activity/connectedness, or multivariate with entries reflecting the email rate for each of the participants. $y_t$ would be the infrequently observed actual count, so $A$ would be a unit matrix.

By assuming multivariate normality of $x_t$, $\epsilon_t$, $\delta_t$, as well as some identifying constraints on the covariance matrices of these variables (or vectors -- e.g., independence of $\delta_t$ and $\epsilon_t$ from each other and from the history $(x_s,y_s), s=t-1, t-2, \ldots$), you can manipulate the conditional distributions and obtain improved predictions $x_t|\{x_s, y_s, y_t\}$ of the state vector $x_t$. The formulae are provided on the Wikipedia page I linked.

Answer 2

This depends on the nature of the data. Your problem formulation suggests you should reconsider what you're trying to estimate. I'm confused. I don't know why you'd want to estimate email volume in a subset of persons that vary with time. You could use a Poisson regression model using offsets to estimate the individual rate of email return, assuming you've formulated this problem correctly.

Turning to the problem of estimating a state-space model in the presence of missing data:

Using a regression modeling approach, you can estimate a lagged effects model by using any number of previous states as regressors in the model. The number of such states required depends on the stationary nature of the process. This can be estimated using cross validation in the modeling process, though. If X is a polytomous model, consider a multinomial regression model, if X outcomes are continuous, consider a linear model.

Having a regression modeling approach allows us to use methods for handling missing data. First off, you can ignore missing data (complete case analysis) and estimate a consistent, albeit inefficient, model. Multiple imputation and EM algorithms are alternate approaches which give modest improvements in efficiency. Note: in multiple imputation, one can use future states to impute past states. For an overview of missing data methods in regression modeling, consult Rubin, "Statistical Analysis with Missing Data".