I am reading in historical sensor data from a plant. I found out that there are intermittent periods where between time t1 and time t2, the data points are linearly interpolated. I came to know, that this is done by the server automatically, when the data is missing between t1 and t2 (eg. sensor being off etc.). I am providing an example below, where the data goes missing between A,B. Manually filtering out these unwanted data points improves my model quality by quite a lot. I would like to know what is a smart approach to filter out such data points apart from checking the slope ? enter image description here


Perhaps more of a programming problem than a statistical one, but there is a simple solution: Write an algorithm that fits a line across a moving window. Wherever the line has a perfect fit, the data has likely been interpolated. How likely? That depends on the number of digits the sensor records and the size of the moving window.

For example, a window of $100$ ms (? I can't see what your unit of measurement is in the question), appears to have quite a lot of variation going on in the non-interpolated sections, but is perfectly linear where interpolated, judging from the figure.

Of course, the larger the window, the larger the minimum size of the interpolations it will be able to pick up (and thus the fewer it will pick up), but the smaller the window, the more likely it will pick up coincidental cases.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.