Can I compare Mahalanobis distances from different distributions?

Question

I have a multivariate dataset representing multiple locations, each of which has a set of reference observations and a single test observation. For each location, I would like to measure how anomalous the test observation is relative to the reference distribution, using the Mahalanobis distance. Then I would like to compare these Mahalanobis distances to evaluate which locations have the most abnormal test observations.

My question is: is it valid to compare Mahalanobis distances that were generated using different reference distributions? I have only ever seen it used to compare test observations relative to a single common reference distribution. The data for each of my locations is structurally identical (same variables and number of observations) but the values and covariances differ, which would make the principal components different for each location.

I have a good intuitive understanding of how Mahalanobis distance works by combining PCA and SED, but am not a statistician so would appreciate answers on the less technical end of the spectrum.

Answer 1

Under normality the Mahalanobis distance of a new observation $y$ from a sample characterised by mean $\bar x$ and sample covariance matrix ${\bf S}$ is an $F_{p;n-p-1}$-distribution. For large enough $n$ this can be approximated by a $\chi^2_p$-distribution. These distributions only depend on $n$ and $p$ (the asymptotic $\chi^2$ even only on $p$), not on the true mean and covariance matrix, and distances can therefore be compared over samples from different populations. Obviously variation can be large if $n$ is relatively small to estimate mean and covariance matrix for relatively large $p$.

If the distributional shapes are not normal, this will not normally hold, although if the distributional shapes are the same for all your different populations, I'd still expect that the distribution of Mahalanobis distances is independent of the location and not strongly dependent on the underlying dependence structure, so I'd say comparison is still OK, although over-interpretation should be avoided (it's something of a hack) and outliers may spoil comparability. Also if distributional shapes are highly skew, an essentially symmetric way of assessing anomality as the Mahalanobis distance may be problematic.

If distributional shapes clearly differ between the different populations, by and large Mahalanobis distances may still bring forth roughly similar value ranges and may very roughly be comparable, but I'd be reluctant to say that they measure "anomality" in a properly comparable way, as small Mahalanobis distances may for one distribution indicate that an observation is really in the high density core, whereas another distribution may have low density where Mahalanobis distances are lowest (although high Mahalanobis distances will always indicate anomality).

Answer 2

The Mahalanobis distance is a measure of distance that can be seen as a multidimensional equivalent of the one-dimensional standardized distance (for instance standard score). Maybe, when you are thinking of this one-dimensional concept (which is ubiquitous) you can get a better intuitive view whether this type of comparison is valid. (a difference is of course that in the one-dimensional case there is an explicit direction that might be meaningful)

The justification of the value depends on the situation, whether it makes sense to standardise the distance before comparison. Examples where standardized distances are used are results from IQ tests. These are always standardized based on references that are different for different groups (mostly age is used as the difference). Another example is a growth chart where the growth of a child is compared and expressed as a quantile relative to a sub-population (kids of the same age, gender and country). Outlier detection is another example (and a case where direction is less important).

So there are motivations that justify comparison of a standardized score. A statistical issue with the validity of such comparisons would relate to the accuracy of the estimates. Due to errors in the estimate of the population mean and variance (and in the multidimensional case covariance-matrix), the comparison of values may be more or less meaningful depending on the random variations in these estimates. (To take a previous example, IQ-scores, for younger kids IQ scores are much more variable than for adults, so you can't compare the IQ scores meaningfully when the age gap is large)