I'm currently working on a machine learning project. I have data from over 50 sensors but the time at which data was recorded are not the same, the data is not synchronized and there are missing data.

Most of the sensors record data at an interval of 10 minutes, one record at an interval of 5 minutes and another record at an interval of 1 minute.

What is the best approach to pre-process the data for machine learning? I'm trying to predict energy consumption which is being recorded at an interval of 1 minute.

  • $\begingroup$ Welcome to Cross Validated. What exactly is the goal of your analysis? If we don't know what you want to do, we cannot possibly say whether the form of the data is sufficient or if it requires more processing. $\endgroup$ – deemel Dec 26 '19 at 9:47
  • $\begingroup$ I'm trying to predict energy consumption which is being recorded at an interval of 1 minute. $\endgroup$ – Y. Gopee Dec 27 '19 at 6:07

Let's assume the underlying physics of your system of interest permits us to consider the last sensor records to be the most relevant record to each record of your target (energy consumption/ 1 min interval sensor). You can create a dataframe in which every row consists of 1 min interval energy consumption record as the target and other sensors records filling other columns (predictors). Now, since they are not synchronized, you could repeat the latest of the 5 min and 10 min sensors readings into every row until there is a new reading. basically it will be filling the gaps until there is none. See schematic picture below. The only way to see if this makes sense would be to understand all physical parameters of your system. It may very much be the case that the last reading of the sensors are not relevant and that all data points need to be thrown out except the points at which there are synchronous original readings from all sensors. Again domain knowledge would dictate which way makes more sense.

enter image description here

  • $\begingroup$ This approach seems good enough when I have missing data for like 30 minutes to 1 hour. What can I do when in some case data are missing for 6-48 hours? The data from these sensors change throughout the day, they are weather/environment variables. $\endgroup$ – Y. Gopee Dec 27 '19 at 8:13
  • $\begingroup$ I would remove all the data points in which there are large gaps and then restart adding the rest of the data (basically cut and append the data frame). After the 48 hr gap is removed, remove an additional next hour (or any other time frame depending on the response time) as a "buffer" then keep the rest of the data. $\endgroup$ – Kinformationist Dec 27 '19 at 18:53

Not the answer you're looking for? Browse other questions tagged or ask your own question.