1
$\begingroup$

I am working on a stream learning of a univariate timeseries data collected from a sensor every hour with the seasonality of a week. The input data might be missing for more than a week completely at random because of sensor malfuncion. Therefore, I need a strategy to impute the missing data before the data is given to the actual stream learning algorithm.

I thought of following solutions :

  1. Use Averaging or Kalman filters for filling. But they are not good for filling up large set of continuous missing points.

  2. Use the model prepared by forecasting algorithm to generate forecasted data. Use this data to impute the missing data. And feed the same imputed timeseries to forecasting algorithm. Can this approach cause trouble?

Any pointers for imputation will be helpful for stream learning.

$\endgroup$
  • 1
    $\begingroup$ "Use Averaging or Kalman filters for filling. But they are not good for large filling large missing data." -- are you implying that there are a lot of "random" missing values in a given section of time? Example: a whole hour is missing where there should be readings every minute. If there is data missing for an entire week then the data may not be MCAR. But, more context is needed. $\endgroup$ – Jon Aug 8 '16 at 18:06
  • $\begingroup$ Sorry, I forgot to add that the data is collected from the sensors once every hour. There are situations when the sensor is not functioning (eg: communication problems or power failures) and hence no data at all for more than a week. $\endgroup$ – chowta Aug 9 '16 at 8:43
  • $\begingroup$ I reframed the sentense about kalman filters. I hope this makes sense now $\endgroup$ – chowta Aug 9 '16 at 8:53
  • 1
    $\begingroup$ If there is an underlying patter/reason why the sensors fail, then the data may be MAR; frequency of sensor failure should be taken into account if it occurs regularly. That aside, for a week (7*24 sensor readings) the missing data would not be easy to fill in. A rolling average or kalman filter would not be helpful for a big window of missing data like that. If the other sensors are very similar, I would either bootstrap, or find a mean/median of readings across all sensors to fill in for the missing data points during the specific window of time. $\endgroup$ – Jon Aug 9 '16 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.