Interquartile range finding more than 10 times outliers than zscore

Question

I'm learning about outlier detection and I wrote these two methods to get the row indexes of the instances that have outliers so I can drop them later. The problem is I'm getting two numbers very far from each other with these two methods.

I would like to know if interquartile range tends to identify more than zscore or maybe the code is wrong. The dataset I'm using is "Rain in Australia" and "att_num" is only the numerical attributes.

indexes = att_num.index
indexes_to_remove = set()

# interquartile range
indexes_to_remove.clear()
for name in att_num:
  q1 = att_num[name].quantile(0.25)
  q3 = att_num[name].quantile(0.75)
  iqr = q3 - q1
  x = 1.5 * iqr
  lower_limit = q1 - x
  upper_limit = q3 + x
  condition = ((att_num[name] < lower_limit) | (att_num[name] > upper_limit))
  indexes_to_remove.update(indexes[condition].tolist())
print("Lines to remove: {}.".format(len(indexes_to_remove)))

Lines to remove: 71176.

.

# z-score
indexes_to_remove.clear()
for name in att_num:
  std_unit = 3
  #scores = ((att_num[name] - att_num[name].mean()) / att_num[name].std())
  scores = stats.zscore(att_num[name])
  condition = ((scores < -std_unit) | (scores > std_unit))
  indexes_to_remove.update(indexes[condition].tolist())
print("Lines to remove: {}".format(len(indexes_to_remove)))

Lines to remove: 6975

Answer 1

Yes - using the IQR method can find more "outliers" than using the standard deviation (or zscore). It depends on the distribution of the data. Data that's peaked with long tails will have a comparatively low IQR, so the IQR method will find lots of outliers.

While both these methods can be used for outlier detection when considering only one variable, using them to detect outliers across multiple variables doesn't work well. In your code, you are flagging any sample that is found to exceed the limit in any one attribute as an outlier. If you've got several independent numeric attributes you will end up flagging quite a high proportion of your dataset as outliers. Is that reasonable? Is having >70,000 (or even only ~7000) outliers what you would expect for your dataset?

If you're interested in other methods of outlier detection that may be more appropriate for your dataset, see What is the best way to identify outliers in multivariate data?, the answers suggest a few possible approaches, or Sergen Cansiz's Multivariate Outlier Detection in Python in the "Towards Data Science" blog for an example of how to detect outliers using Mahalanobis Distance.