0
$\begingroup$

I'm trying to calculate a robust z score, and I'd like to understand the constants I'm using, and their impact on my statistic.

One corner case I've noticed is when my sample happens to be all the same value, since there is no dispersion, both a regular z-score, and a MAD-based robust z-score have zero std/MAD in the denominator, and produce undefined results. Even if quite mathematically reasonable, this is is less robust than I'd hope (it seems maybe more intuitive to smooth this to the constant itself than to undefined).

A definition I saw from an IBM product recommended special-casing the denominator for this reason.

Coding their suggestion in pandas for example, on a rolling basis, I might have something like this:

def rolling_robust_zscore(d, window=5):
  num = d - d.rolling(window).median()
  meanad = num.apply(np.abs).mean()
  mad = num.apply(np.abs).median()
  denom = np.where(mad, 1.48258 * mad, 1.253314*meanad)
  return num / denom

My questions are:

  • What are the values they provided, 1.48258 for the median absolute deviation and 1.253314 for the mean absolute deviation? How were these chosen?
  • Is this robust estimator good, statistically? Do these particular values introduce a bias or other undesirable properties?
  • Is there a better choice for a robust rolling z-score?
$\endgroup$
2
  • 1
    $\begingroup$ I would guess "1.48258" is an approximation to $1/\Phi^{-1}(3/4)= 1.48260\ldots.$ $\endgroup$
    – whuber
    Aug 15, 2021 at 20:48
  • $\begingroup$ same for 1.2533... it's the inverse of the assymptotic value of the mean AD evaluated at a gaussian. $\endgroup$
    – user603
    Aug 15, 2021 at 20:58

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.