How to handle incomplete data in Kalman Filter?

Question

What are some typical approaches to handling incomplete data in the Kalman Filter? I'm talking about the situation where some elements of the observed vector $y_t$ are missing, distinct from the case where an entire observed vector $y_t$ is missed. Another way of thinking about this would be that the dimension $p$ of the observed vector is different for each time point.

To explain my context a bit further, the observations are estimated parameters from a logistic regression performed at each time point. While each logistic regression includes the same covariates, sometimes the estimates are undefined due to collinearities in the data for that time point.

Answer 1

What is needed is simply to have a variable observation matrix, i.e. in the observation equation: $$ \boldsymbol{Y_t} = \boldsymbol{A_t}\boldsymbol{\theta_t} + \boldsymbol{R_t}\boldsymbol{e_t} $$ matrix $\boldsymbol{A_t}$ (and $\boldsymbol{R_t}$) should omit at time $t$ the rows corresponding to NA entries in $\boldsymbol{Y_t}$. Most packages in R, for instance, will take care of that: you can have the observed multivariate time series with NA values without problems.

Answer 2

The simplest solution is to just use any measurement value (the last good one is best), but set the corresponding measurement noise variance to an extremely large number. In effect, the fake measurement will be ignored. The Kalman filter is balancing measurement uncertainty against model uncertainty, and in this case, you're just estimating based on whatever the state model predicts plus other measurement corrections. As long as the measurement is unavailable, any states that would become unobservable without that measurement would have their uncertainty grow over time because of process noise. That's very realistic - your confidence in projections based on old measurements continually decreases over time. (This is true for this solution or for the case of temporarily changing the filter structure to eliminate the measurement).

This formulation assumes you're using a Kalman filter that updates both the state and covariance matrix at each step, not the steady state version. This is the simplest approach if your software doesn't already have special handling for unavailable values. (And software that does have missing value handling might well handle it this way). This approach in theory should accomplish exactly the same thing as modifying the measurement matrix size and measurement covariance matrix size. A measurement with almost infinite variance contributes the same information as no measurement at all. But this way, there's no need to change the structure of the filter or store all the possibilities - it's just one parameter change (assuming the typical case of each measurement noise error being independent, so that the measurement covariance matrix is diagonal).