Skip to main content
Cross Validated

Return to Question

edited tags
Link
kjetil b halvorsen ♦
kjetil b halvorsen ♦
  • 82.8k
  • 32
  • 201
  • 663
deleted 1 character in body
Source Link
Nick Cox
Nick Cox
  • 59.5k
  • 8
  • 136
  • 212

I'm a programmer with a small statistics background and I need to find outliers in a small list of integers and floats.

After some search on google I found the Iglewicz and Hoaglin outlier test witchwhich creates a modified z-score Mi for every value in the list and check it against an threshold (normally 3.5).

$$M_{i} = \frac{0.6745(x_{i} - \tilde{x})} {\mbox{MAD}}$$

I wrote a litte python script to test it. At first it worked great, but after a few tests I spotted an error.

If you try to find outliers (with my script) in an list with many identically values and one outlier e.g. data = [10, 10, 10, 10, 10, 10, 10, 100] the MAD(median absolute deviation) becomes 0 and this leads my to my question: "What should I do if the MAD becomes 0?".

My first idea was to set the MAD to ∞, but this causes the script to find no outlinersoutliers.

My second idea was to add very small offsets to the values to make them unique e.g. data = [10.0, 10.00000001, 10.00000002, 10.00000003, 10.00000004, 10.00000004, 10.00000005, 100]. This way the MAD can't become 0 and my script is able to detect the outlier 100.

Does somebody have better ideas?

Am I doing something wrong?

I'm a programmer with a small statistics background and I need to find outliers in a small list of integers and floats.

After some search on google I found the Iglewicz and Hoaglin outlier test witch creates a modified z-score Mi for every value in the list and check it against an threshold (normally 3.5).

$$M_{i} = \frac{0.6745(x_{i} - \tilde{x})} {\mbox{MAD}}$$

I wrote a litte python script to test it. At first it worked great, but after a few tests I spotted an error.

If you try to find outliers (with my script) in an list with many identically values and one outlier e.g. data = [10, 10, 10, 10, 10, 10, 10, 100] the MAD(median absolute deviation) becomes 0 and this leads my to my question: "What should I do if the MAD becomes 0?".

My first idea was to set the MAD to ∞, but this causes the script to find no outliners.

My second idea was to add very small offsets to the values to make them unique e.g. data = [10.0, 10.00000001, 10.00000002, 10.00000003, 10.00000004, 10.00000004, 10.00000005, 100]. This way the MAD can't become 0 and my script is able to detect the outlier 100.

Does somebody have better ideas?

Am I doing something wrong?

I'm a programmer with a small statistics background and I need to find outliers in a small list of integers and floats.

After some search on google I found the Iglewicz and Hoaglin outlier test which creates a modified z-score Mi for every value in the list and check it against an threshold (normally 3.5).

$$M_{i} = \frac{0.6745(x_{i} - \tilde{x})} {\mbox{MAD}}$$

I wrote a litte python script to test it. At first it worked great, but after a few tests I spotted an error.

If you try to find outliers (with my script) in an list with many identically values and one outlier e.g. data = [10, 10, 10, 10, 10, 10, 10, 100] the MAD(median absolute deviation) becomes 0 and this leads my to my question: "What should I do if the MAD becomes 0?".

My first idea was to set the MAD to ∞, but this causes the script to find no outliers.

My second idea was to add very small offsets to the values to make them unique e.g. data = [10.0, 10.00000001, 10.00000002, 10.00000003, 10.00000004, 10.00000004, 10.00000005, 100]. This way the MAD can't become 0 and my script is able to detect the outlier 100.

Does somebody have better ideas?

Am I doing something wrong?

Source Link
szuuuken
szuuuken
  • 213
  • 2
  • 6

Iglewicz and Hoaglin outlier test with modified z-scores - What should I do if the MAD becomes 0?

I'm a programmer with a small statistics background and I need to find outliers in a small list of integers and floats.

After some search on google I found the Iglewicz and Hoaglin outlier test witch creates a modified z-score Mi for every value in the list and check it against an threshold (normally 3.5).

$$M_{i} = \frac{0.6745(x_{i} - \tilde{x})} {\mbox{MAD}}$$

I wrote a litte python script to test it. At first it worked great, but after a few tests I spotted an error.

If you try to find outliers (with my script) in an list with many identically values and one outlier e.g. data = [10, 10, 10, 10, 10, 10, 10, 100] the MAD(median absolute deviation) becomes 0 and this leads my to my question: "What should I do if the MAD becomes 0?".

My first idea was to set the MAD to ∞, but this causes the script to find no outliners.

My second idea was to add very small offsets to the values to make them unique e.g. data = [10.0, 10.00000001, 10.00000002, 10.00000003, 10.00000004, 10.00000004, 10.00000005, 100]. This way the MAD can't become 0 and my script is able to detect the outlier 100.

Does somebody have better ideas?

Am I doing something wrong?