2
$\begingroup$

I've read in some papers (such as this) and CrossValidated questions (such as this, that people are using mahalanobis distance based on robust estimations of location and scatter using minimum covariance determinant method.

Usually, they run the fastMCD algorithm to obtain estimates for location and scatter, center the data using this robust estimate and calculate the mahalanobis distance, then they compare the distances to the chi2 distribution, and whatever would be unlikely under the chi2 is treated as an outlier.

However, if I follow the approach outlined in e.g. this answer or this sklearn tutorial, it would seem that if my robust location estimate is not equal to the sample mean, then the centered data won't be zero mean, in which case the comparison to chi2 distribution won't work, because all my observations will have distance centered around, say, 185.

Is my understanding correct, or am I missing something? If I am right, what would then be the right approach?

$\endgroup$
1
  • 2
    $\begingroup$ The comparison to chi-squared is a (very rough) approximate rule of thumb in the first place. If you are concerned about its application in a particular setting, experiment! $\endgroup$
    – whuber
    Commented Sep 5, 2022 at 16:17

1 Answer 1

3
$\begingroup$

Robust estimators are meant to work with populations that have some "good" data and a certain percentage of outliers. They are meant to be influenced very little or not at all by outliers.

Let's say you have a certain big enough proportion of observations from the multivariate normal distribution, and otherwise outliers. The mean will then be affected by the outliers; and to the extent that the MCD location estimator ignores the outliers, it will estimate the theoretical mean of the normal distribution in fact better than the arithmetic mean. The key here is that the "correct reference" for outlier detection is the mean of the non-outliers, not the overall mean! The MCD estimator estimates the former, the arithmetic mean the latter. If indeed the non-outliers are normal, and if indeed the outliers are placed so that the MCD can ignore them, the $\chi^2$-approximation will be fine for the non-outliers, and better for MCD than for the arithmetic mean.

If there are no outliers or the outliers are elliptically distributed, the MCD estimates the same location as the arithmetic mean, so they will be approximately the same, and $\chi^2$ will still be alright, even though the MCD will be somewhat less precise. The $\chi^2$-approximation will be slightly worse for clean normal data, but may be better than for the arithmetic mean if there are outliers, even if elliptically distributed. If the underlying distribution isn't normal and not even normal plus outliers, $\chi^2$ will be in trouble regardless of whether you use the arithmetic mean or the MCD.

Note though that the normal model can be seen as a calibration device for outlier identification rather than as an "assumption", which means that using the $\chi^2$ for outlier identification doesn't necessary require that normality holds (at least for the non-outliers), but rather that what it means to be "outlying" is defined relative to the normal distribution, the underlying truth being normal or not.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.