Underneath is a picture of a sensor measuring the fill rate of a container on an hourly basis. It goes up to 100% and is then emptied. There is some natural deviation of the sensor due to temperature etc. In theory, the fill rate can only increase up to a certain level and then drop to 0 % because it is emptied.
In practice, we see some hourly fluctuation in the signal because of measurement errors. Examples of this are indicated in yellow on the figure. How do you clean this signal in such a way that it represents a container being filled and emptied? For example, it is impossible that at 7AM the container is 50% full and one hour later it is only 47%. Another flaw in the data is a sudden signal loss of a few hours.
The reason for cleaning this data lies in analytical purposes: descriptive (at what time during the day is this container mostly being filled, how long does it take, ...) and predictive (when will this container be full next time?)
Additional info: cleaning I already applied
- Moving average: still gives a distorted signal.
- If-then rules: by applying some rules about what the signal cannot do, we can do some cleaning. An example of an implemented rule is that there is no negative fill rate change under 30% possible. This leads to a proper cleaning procedure but requires a lot of business insights and fine-tuning. This is also not very robust, hence the need for a more statistical procedure.