Outlier detection using Medial Rule

Question

I have a data set captured every minute. Using Tukey's test for outlier detection (i.e., if number points above Q2+2.3IQR is above 10% of total distribution) then I mark the one minute window as anomalous. It works fine for the first minute. In reality, the system contines the behavior for the next few minutes and the algorithm will not mark the next few windows as anomalies as the median is shifted way above and continuous.

By domain knowledge, the system continues in anomalous state for next two to three minutes and then settles down.

To my understanding, I need to add one more algorithm along with Tukey's test so that the shifts are also measured and marked as anomalies

Any help on finding an algorithm is much appreciated.

PS: I couldn't attach as I don't have enough reputation.

Let me shed more light... My problem is something different from masking.

Window 1:

Distribution: Right tailed normal distribution Outlier based on Median Rule: Yes, as certain number of points lies above Q2+2.3IQR Median of the distribution = 23

Windows 2:

Distribution: Normal distribution but little large spread, as the effect of window 1 continuous in window 2 Outlier based on median rule: No, the spread (variance) of distribution is more and there are no point lies outside Q2+2.3IQR Median of the distribution = 40

Windows 2:

Distribution: Normal distribution but little large spread, as the effect of window 1 continuous in window 2 and 3 Outlier based on median rule: No, the spread (variance) of distribution is more and there are no point lies outside Q2+2.3IQR Median of the distribution = 60

Suddenly the the behavior drops to something like it was in window 1, From the domain knowledge the window 2, window 3 are anomalies. On the other hand we can argue that this cannot be treated as anomalies as the system behavior repeats such a way in learning phase

Answer 1

It appears that you're using the Median Rule rather than Tukey. Have you looked at this paper by Songwon Seo, or something similar? I believe part of your issue is called "masking", and is addressed in this paper by Wang and Serfling.

To be clear, you're saying that the problem is that outliers come in clusters and only the first measurement is marked as anomalous because it throws the IQR off so badly that the next couple of outliers aren't flagged? Could you simply not include the outlier in the calculation of the IQR?

P.S. as far as I can tell, unless you have a very good model of your process, outlier detection is a bit of a voodoo art.