# Proper Imputation and bias-correction on degrading signal with Kalman Filtering?

A signal degrades in its quality. Some signals are far more robust to degradation while others are not. We will simulate degradation by randomly removing values from a function and then applying Kalman-filter on the missing values. Some functions such as Sin are very bad to random degradation, losing the trend very quickly and joining subcycles together. Then again some signals are far more robust to random degradation which I want to understand better.

If the end-signal always end up to about the same signal, how can you describe degradation-prone signal? Robust?

Helper questions

1. Which Kalman filter should you build to correct the signal properly?

2. What would be the proper imputation techniques to correct the randomly-removed values? Structural timeseries with R's DLM package or ImputeTS::na.kalman (this looks bad as demonstrated in the example below)?

3. The bias-correction with the Kalman -filtering, whatever chosen, introduces some bias, how to quantify that?

Simple Example about degrading signal corrected with ImputeTS::na.kalman where corrections not good

We show a signal that can preserve the shape only very little, namely Sin fuction.

library(imputeTS)

si<-sin(1:4000/100)
for(xx in 1:5) {
is.na(si) <- sample(length(si), 0.7*length(si))
plot(na.kalman(si)); Sys.sleep(2)
}


Degradation of the trend is already obvious, two periods joined together after the K-filtering

Degradation of the timeseries is more serious, many trends joined together, very bad robustness to degradation with Sin function

• I'm probably a bit slow but don't really understand what your question is about. – Jarle Tufto Oct 11 '18 at 15:17
• @JarleTufto I clarified the question, does it make more sense now? – hhh Oct 12 '18 at 7:41
• Probably way too late, but did you try the different parameter options of the na_kalman function, see here: rdocumentation.org/packages/imputeTS/versions/3.0/topics/… – stats0007 Nov 18 at 17:58

I am unable to answer the question but I provide some related material on Kalman filtering, that are less prone to numerical instabilities. I am not sure yet whether the instability of the signal is due to the package so trying other packages to filter the signal to see what happens may help.

Possibly related concepts

• detectablity
• stabilizability
• the rank of the controlability / observability matrices
• Gramians
• timeinvariant control

1. I would have a look at DLM vignette since "naive implementations of the Kalman filter are known to suffer from numerical instability for general DLM" that is designed particularly for Bayesian MCMC applications.