# apply Kalman smoothing to irregularly spaced time series

My data is an irregularly spaced time series:

        date    adate
0   2012-03-30  0.0
1   2012-03-30  1.0
2   2012-03-31  19.0
3   2012-04-19  1.0
4   2012-04-20  1.0
... ... ...
240 2019-11-08  6.0
241 2019-11-14  0.0
242 2019-11-14  1.0
243 2019-11-24  13.0
244 2019-12-07  NaN


since I want to perform some sort of timeseries analysis on the data (ARIMA , preferably) I want to interpolate it such that every data point is evenly distributed. I have read I can apply Kalman smoothing a series of data sampled at irregular time points. I have read a few papers and have found a number of libraries to apply Kalman filters like pykalman but I haven't understood how to apply it simply , like you can apply a linear or cubic interpolation using scipy/ pandas.

• What you have there is not an irregularly spaced time series because you have multiple observations for a single point in time (e.g. 2019-11-14). Aside from that, you don't need to interpolate with Kalman smoothing first; that would involve fitting a state space model which can just be an ARIMA model anyway. If you were to fit another ARIMA model after Kalman smoothing you would also distort the dynamics because information is leaking from the future. Feb 24 '20 at 11:51
• I understand. If that is the case, how would you suggest i make this time-series into an evenly spaced one? Say, after removing the datapoints that have multiple observations for a single date.? Feb 25 '20 at 6:03
• I suggest you TSMOOTHIE: a python library for timeseries smoothing and outlier detection in a vectorized way github.com/cerlymarco/tsmoothie. TSMOOTHIE provides a robust implementation of Kalman smoothing Aug 24 '20 at 16:19

Setting aside the repeated measures for now, the easiest way to deal with an irregularly spaced time series with relatively regular "small" gaps is to view it as a regularly spaced time series with missing data. Here, since your smallest gap is 1 day, you can consider it as daily data but with some days missing:

date    adate
2012-03-30  1.0
2012-03-31  19.0
2012-04-01  NA
2012-04-02  NA
...         ...
2012-04-18  NA
2012-04-19  1.0
...         ...


The situation is a little bit different if you have a very large variance in the size of the gaps, for example if you had millisecond-level time stamps but sometimes go a whole year without any observation; in that case it can be handled more efficiently in another way (e.g. by having time-varying matrices in the state space model used by the Kalman filter).

The Kalman filter will allow you to fit an ARIMA model with missing values by computing the likelihood which you can then optimize over the parameters. You can then use that model to forecast. If you need, you can also use the Kalman filter or smoother to get the distribution of the missing values conditional on your data (only past data for the filter, or including future data for the smoother) and parameters.

But you do not need to impute these values first, and doing this is not a preliminary step to an analysis (it is the analysis, you have already picked an ARIMA model at this point).

As for the repeated measures, if it makes sense for the domain you can sum or average those values on a given day. If it doesn't and you have no way to differentiate those records in a given day, you can set up a state space model where the state is, for example, given by:

$$X_t = \phi X_{t-1} + \eta_t$$

And the observation equation is:

$$Y_t^{(i)} = X_t + \varepsilon_t^{(i)}, i = 1, ..., n_t$$

This would be an ARIMA(1,0,0) model with repeated measures of varying sample sizes depending on the day. The Kalman filter can accommodate state space models with varying observation dimension.

• I understand. Linearly interpolating my data in a walk forward validation technique, gives me forecasts that are really perfect. which I presume is because of the linear interpolation of the missing datapoints. Hence , I believe kalman smoothing would be a better option, but I haven't been able to apply a smoother to my data. Feb 26 '20 at 9:27