1
$\begingroup$

What's the best way to find outliers in the time-series, encountering continuity?

I attached two time-series that I'm interested to filter. One is less noisy, and one is a bit noisier. I'm mostly interested in the first one.

BTW the data looks like it is periodic, however, it is not and it is quite short (200-500 samples).

Also, I don't know much about the process

Thanks

ts 1 ts 2

$\endgroup$
6
  • $\begingroup$ Because these look periodic, and even a very short term quasi-period could be extremely helpful for improving an outlier detection method, could you be more specific about how they are expected to depart from periodicity? $\endgroup$ – whuber Sep 22 '20 at 16:43
  • $\begingroup$ It is controlled by humans. In most cases human $\endgroup$ – 124bit Sep 22 '20 at 22:14
  • $\begingroup$ That doesn't seem like a necessary reason for it to be non-periodic. How does the human control manifest itself in the data behavior? Can you post an illustration? $\endgroup$ – whuber Sep 22 '20 at 22:16
  • $\begingroup$ Excuse me, I posted the previous short answer accidentally. This process is controlled by a human and has many degrees of freedom. A human can tune it to have a period from 0.5 to 4 seconds. Also human controls shape, but the shape of each period always looks like a hill. However, I thought such kind of task has some common approaches even without taking in account period and shape. Just continuity. Isn't it? Thanks $\endgroup$ – 124bit Sep 22 '20 at 22:22
  • $\begingroup$ The more you can assume (or even know) about the underlying process or the likely behavior of the series, the better able you will be to analyze it. Applying some generic technique that assumes little or nothing is possible, but is not going to work well on examples like your second series. $\endgroup$ – whuber Sep 23 '20 at 13:00
1
$\begingroup$

The best way is not to filter "outliers" at all

What we call "outliers" in statistical analysis are points that are distant from the majority of the other points in a distribution. Diagnosis of an "outlier" is done by making a comparison to an assumed distributional form, and statistical tests for outliers compare the position of the outlier to what is expected as a maximum deviation under the assumed distribution. Outliers are sometimes caused by measurement error (i.e., recording a data point incorrectly) but usually they are valid observations that just happen to be in the "tails" of the relevant distribution. Often the diagnosis of outliers occurs when the data follows a distribution with high kurtosis (i.e., fat tails), but we compare the data points to an assumed distributional form with low kurtosis (e.g., the normal distribution).

Thus, if we are doing statistical analysis properly, when we identify "outliers" in the data, this means that we have identified that the underlying assumed distributional form does not have sufficiently fat tails to properly describe the observed data. Unless we have reason to believe that measurement error has occurred (in which case we might legitimately filter out invalid data), this is a deficiency of the model, not the data. Unfortunately, some analysts prefer to discard aspects of reality that do not conform to their models, rather than discarding models that do not conform properly to reality. In doing so they engage in the fallacy of "cherry picking".

In regards to your time-series analysis, the best method of analysis here would be to first fit the periodic parts of the data and then examine the residuals to see what kind of distribution accommodates their shape well. The variance of the residuals is clearly related to the periodic part of the data, so you will also need to accommodate for this in your model. Once you have a set of residuals from fitting the periodic parts of the model, you will be able to take a crack at forming a sensible underlying distribution for the "error term". This might involve using a transformation of scale to (implicitly) deal with skewness or kurtosis of the residuals. Once you have a good model, it should describe the periodic behaviour of the time-series well, and it should also have an error distribution that fits with the residual data. Ideally, if you conduct an "outlier" test on your model, it will pass the test (i.e., it will not identify points that are too far out in the tails for the model form to be plausible).

$\endgroup$
1
  • $\begingroup$ Thanks a lot. The shape characteristics of data are obvious, however, the period may significantly change during the series (0.5 to 4). As I wrote in question, from my perspective it is not strictly periodic. Does your advice regarding the fitting of periodic parts applicable in such circumstances? $\endgroup$ – 124bit Oct 6 '20 at 23:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.