1
$\begingroup$

I thought of this way of detecting outliers. What are the "bad" properties of this method?

For example, say you have a time series, and you want to check if the latest observation is an outlier.

Firstly, I limit the time series to the past N elements, so that the time series are always of the same length, and data from way in the past is ignored.

Then I calculate the standard deviation with AND without the latest observation.

I then flag it as an outlier if that change is big, i.e. if $$(\text{std_with} - \text{std_without}) / \text{std_with}$$ is a big percentage.

Is this method bad? What are your criticisms?

Improve this question
$\endgroup$
6
  • $\begingroup$ Seems ok. Maybe add some sort of statistical test for significance of the difference? Otherwise you'd have to subjectively define "big change" $\endgroup$
    – PaulG
    Dec 15, 2020 at 16:12
  • $\begingroup$ Right, defining a "big" change is the challenge here, as it will depend quite a bit on the N. If you have lots of observations, the addition/deletion of a single datapoint won't change the SD by much no matter how much of an outlier it is. Conversely, using smaller N will result in larger changes by chance alone. $\endgroup$
    – Nuclear Hoagie
    Dec 15, 2020 at 16:18
  • $\begingroup$ This is a standard method (at least when the "big percentage" is chosen carefully)--but why make up your own method when you can read about them and find open source implementations? $\endgroup$
    – whuber ♦
    Dec 15, 2020 at 17:40
  • $\begingroup$ Read about them where? $\endgroup$
    – amen
    Dec 15, 2020 at 17:57
  • $\begingroup$ You need to be clear about what you mean by "outliers" and what your objectives are in identifying them. Here are a few places where you can read about them: stats.stackexchange.com/questions/259654/… and stats.stackexchange.com/questions/15497/… $\endgroup$
    – Michael Lew
    Dec 15, 2020 at 20:06

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.