10
$\begingroup$

What are some typical approaches to handling incomplete data in the Kalman Filter? I'm talking about the situation where some elements of the observed vector $y_t$ are missing, distinct from the case where an entire observed vector $y_t$ is missed. Another way of thinking about this would be that the dimension $p$ of the observed vector is different for each time point.

To explain my context a bit further, the observations are estimated parameters from a logistic regression performed at each time point. While each logistic regression includes the same covariates, sometimes the estimates are undefined due to collinearities in the data for that time point.

Improve this question
$\endgroup$
2
  • $\begingroup$ I suggest you this simply python implementation stackoverflow.com/a/63630858/10375049 $\endgroup$
    – Marco Cerliani
    Aug 28 '20 at 9:37
  • $\begingroup$ @MarcoCerliani, the question you linked to deals with missing time steps, rather than incomplete data for a given time step, right? $\endgroup$
    – RTbecard
    Oct 25 '21 at 0:40
7
$\begingroup$

What is needed is simply to have a variable observation matrix, i.e. in the observation equation: $$ \boldsymbol{Y_t} = \boldsymbol{A_t}\boldsymbol{\theta_t} + \boldsymbol{R_t}\boldsymbol{e_t} $$ matrix $\boldsymbol{A_t}$ (and $\boldsymbol{R_t}$) should omit at time $t$ the rows corresponding to NA entries in $\boldsymbol{Y_t}$. Most packages in R, for instance, will take care of that: you can have the observed multivariate time series with NA values without problems.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Would like to bump up this best answer for attention. I read the book Time Series Analysis and Its Applications: With R Examples by Robert Shumway, and in chapter 6, the author described the way of handling missing observations in $Y_t$ is by filling with zeros. How does your method of row/column elimination compare with this method? $\endgroup$
    – Will Gu
    Jan 12 '17 at 17:27
  • $\begingroup$ I think both methods are equivalent. Note, however, that what Shumway & Stoffer propose (and claim it is computationally simpler; I guess it depends on which software you are using) requires filling with zeros not only $Y_t$ but other arrays as well (check their page 347, just above (6.79) of the third edition). $\endgroup$
    – F. Tusell
    Jan 15 '17 at 13:31
  • $\begingroup$ For those who don't have access to Shumway: You have to 0 out the missing observation rows in $Y$ and $A$ (your residual vector and observation model matrix), but replace the missing parts of $R$ (observation covariance matrix) with a component identity matrix (1's along the missing diagonals, and 0's in the missing covariances). $\endgroup$
    – RTbecard
    Oct 26 '21 at 20:37
1
$\begingroup$

The simplest solution is to just use any measurement value (the last good one is best), but set the corresponding measurement noise variance to an extremely large number. In effect, the fake measurement will be ignored. The Kalman filter is balancing measurement uncertainty against model uncertainty, and in this case, you're just estimating based on whatever the state model predicts plus other measurement corrections. As long as the measurement is unavailable, any states that would become unobservable without that measurement would have their uncertainty grow over time because of process noise. That's very realistic - your confidence in projections based on old measurements continually decreases over time. (This is true for this solution or for the case of temporarily changing the filter structure to eliminate the measurement).

This formulation assumes you're using a Kalman filter that updates both the state and covariance matrix at each step, not the steady state version. This is the simplest approach if your software doesn't already have special handling for unavailable values. (And software that does have missing value handling might well handle it this way). This approach in theory should accomplish exactly the same thing as modifying the measurement matrix size and measurement covariance matrix size. A measurement with almost infinite variance contributes the same information as no measurement at all. But this way, there's no need to change the structure of the filter or store all the possibilities - it's just one parameter change (assuming the typical case of each measurement noise error being independent, so that the measurement covariance matrix is diagonal).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.