looking to draw on some of your wisdom around modified z-scores as used for detecting outliers.
As far as I can tell from my research, when a distribution might not be normal (e.g. skewed), a modified z-score is a better indicator of outliers than z-score itself. This is because instead of mean it uses median which is a robust estimator of central tendancy if the distribution is not known to be normal.
I am testing both of these plus some other outlier detection algorithms against a list of values where I know one value is an extreme outlier. To help me I created a small python program that calculates both z-scores and modified z-scores against that list and then uses that to check whether any of the items in the list look like outliers. Basically, I was checking those algorithms would successfully detect the extreme outlier I know is there.
To check the distribution of my data the program creates a box plot, the outlier (value = 200) is clearly evident. For reference, the median of this dataset is 58.
The modified z-score for the outlier value of 200 is only 2.81 and this is substantially lower than the 3.5 for consideration as an outlier, hence it does not get labelled as an outlier. FYI, I used 3.5 because that seems to the most recommended value for the cut-off.
The z-score for the outlier value of 200 is 3.40 and this is above the 3.0 for consideration as an outlier, hence it does get labelled as an outlier. FYI, I used 3.0 as the cut-off as that seemed most popular.
My question is, why does the z-score algorithm detect the outlier in my datasets when the modified z-score algorithm does not? This seems counter-intuitive to me, especially as the outlier is obvious from the box plot.
Here is my python in case that is where I have made an error:
import matplotlib as plt
import numpy as np
from scipy.stats import zscore
def z_score_mod(obs):
# modified z-score = 0.6745(xi – x̃) / MAD
med = np.median(obs)
med_abs_dev = np.median(np.abs(obs - med))
z_score_mod = 0.6745 * ((obs - med) / med_abs_dev)
return z_score_mod
# list of observations with reasonably large outlier value = 200 and index = 13
list_of_obs = [58,71,11,18,90,97,15,53,39,22,62,51,10,200,20,64,94,71,73,18,95,96,92,38,26]
# Convert list of observations to numpy array
array_of_obs = np.array(list_of_obs)
# Create box plot to show the outlier
plt.pyplot.boxplot(array_of_obs)
median = np.median(array_of_obs)
# Calculate modified z-score for each array item
array_of_z_score_mod = z_score_mod(array_of_obs)
# For the modified z-scores generated, determine if any are outliers
array_of_outlier_evals = abs(array_of_z_score_mod) > 3.5
# Is the observation value = 200 at index = 13 an outlier?
print('\r')
print(f'Observation at index position 13 = {list_of_obs[13]}')
print(f'Modified z-score of value = {array_of_z_score_mod[13]:.2f}')
print(f'At modified z-score threshold 3.5 is value an outlier : {array_of_outlier_evals[13]}')
# Calculate z-score for each array item
array_of_z_score = zscore(list_of_obs)
# For the z-scores generated, determine if any are outliers
array_of_outlier_evals_2 = abs(array_of_z_score) > 3
# Is the observation value = 200 at index = 13 an outlier?
print('\r')
print(f'Observation at index position 13 = {list_of_obs[13]}')
print(f'z-score of value = {array_of_z_score[13]:.2f}')
print(f'At z-score threshold 3.0 is value an outlier : {array_of_outlier_evals_2[13]}')
