# Collapse of sampled Mahalanobis distance

Let $$\{ x^{(1)}, \ldots, x^{(M)}\}$$ be $$M$$ samples from a $$n$$-dimensional multivariate Gaussian distribution $$\pi_{X} = \mathcal{N}(\mu, \Sigma)$$.

We recall the definition of the squared Mahalanobis distance $$\delta_{X}$$:

$$\delta_{X}(x) = (x - \mu)^\top \Sigma^{-1}(x - \mu)$$

I am interested in computing the sample squared Mahalanobis distance $$\hat{\delta}_{X}$$:

$$\hat{\delta}_{X}(x) = (x - \hat{\mu})^\top \hat{\Sigma}^{-1}(x - \hat{\mu}),$$ where $$\hat{\mu}$$ and $$\hat{\Sigma}$$ are the sample mean and covariance.

I am typically working in the low-data regime where $$M < n$$.

Independently of the dimension $$n$$, I observe the collapse of the sample squared Mahalanobis distance when $$M < n$$, i.e. all the samples have approximately the same Mahalanobis distance.

First graph: Evolution of the mean of the squared Mahalanobis distance for $$M$$ samples from $$\mathcal{N}(0_n, I_n)$$

Second graph: Evolution of the standard deviation of the squared Mahalanobis distance for $$M$$ samples from $$\mathcal{N}(0_n, I_n)$$

In this case, the sample covariance is not full rank. How can I regularize the sample (squared) Mahalanobis distance?

• One could offer a large number of plausible answers. We need some help here: can you offer some context concerning why you want to regularize it and how that might be applied?
– whuber
May 16, 2023 at 14:24
• I am working on the estimation of the parameters of a multivariate t-distribution from limited samples. Unlike Gaussian distributions, the maximum likelihood estimate for the mean and scale matrix are not known in closed form, but must be obtained from an iterative procedure. The weighting of the different samples $x^{(i)}$ in these sample estimators involves the square Mahalanobis distance of each sample. Given this collapse of the Mahalanobis distance, these parameters are poorly estimated. To get started, I am looking at the regularization of the Mahalanobis distance with Gaussian samples. May 16, 2023 at 14:47
• @Mathieuleprovost If I'm not mistaken, there is no way, in a situation with $M<n$, to tell apart a multivariate t- from a multivariate Gaussian distribution (outlier identification and the like break down for the reasons given in my answer), so why do you want to fit a t rather than a Gaussian? May 16, 2023 at 15:13
• @ChristianHennig, do you have any reference on the indiscernibility of Gaussian and t-dist when $M < n$? May 16, 2023 at 15:39
• @Mathieuleprovost No, it's my intuition. (But my intuition is not too bad in such cases.) May 16, 2023 at 15:45

What you observe there has been proved in this arxiv-preprint by Pires & Branco. It may also be published elsewhere but I haven't seen it.

Note that there is a problem here, which is that the (sample) Mahalanobis distance is intentionally defined to be affine equivariant; it gives all directions in n-dimensional space the same importance for standardisation, and will standardise in any direction according to the variance of observations along that direction. This means that the distance between any two observations is assessed relative to the overall variance/covariance structure ("Covstructure") of the point cloud.

What is shown in the preprint is that if $$n\ge M-1$$, information in the data regarding the Covstructure is so scarce that every observation "on average" determines one dimension worth of variance/covariance entries, and in consequence every observation has the same Mahalanobis distance from every other observation (no observation can be shown as in any direction "outlying" with respect to others, as each of them points into an idiosyncratic direction and none is in any sense "between" others, so to say - note that I'm assuming data in "general position", i.e., spanning an as high-dimensional hyperplane as possible).

This is a consequence of affine equivariance, which could be seen as the very essence of Mahalanobis distance. Arguably, if you penalise it, it will morph into something that is really quite different, and the name Mahalanobis distance may no longer be justified. The resulting distance will have to focus on some directions in $$n$$-dimensional space over some others, when it comes to standardising against the variation of the overall point cloud, which runs counter to the basic idea of Mahalanobis.

For example, Euclidean distance with standardised variables is not affine equivariant and will use the variance along the main coordinate axes for standardisation. Variation in other directions due to correlation will be ignored. If you want to penalise in such a way that standardisation is mainly governed by the main coordinate axes, chances are you'll get something of a mix between the Mahalanobis-distance (which is a constant) and standardised Euclidean, which presumably amounts to being more or less equivalent to standardised Euclidean (depending on how exactly you use it). Note also that even Euclidean distance (with and without standardisation) has characteristics in high-dimensional settings that seem undesirable to many, see Hennig, C.: Minkowski distances and standardisation for clustering and classification on high dimensional data. In: Imaizumi, Tadashi, Nakayama, Atsuho, Yokoyama, Satoru (Eds.) “Advanced Studies in Behaviormetrics and Data Science. Essays in Honor of Akinori Okada”, Springer Singapore (2020), p. 103-118.

• Re "The resulting distance will have to prefer some directions in n-dimensional space over some others." Would that be the case for the standard spherical penalty (adding a positive multiple of the identity matrix)? If so, in what sense? Re "Euclidean distance ... will focus on the main coordinate axes:" How so? Euclidean distance is orthogonally invariant, implying it does not focus on any direction.
– whuber
May 16, 2023 at 15:32
• @ChristianHennig, thank you for your thorough answer and pointing this paper. From a practical perspective, it seems that going from $n-5$ to $n+5$ samples has a drastic impact on the standard deviation of the sample Mahalanobis distance. What can we do "to make things work" with let's say a bit less than $n$ samples? May 16, 2023 at 15:45
• @whuber The standard spherical penalty would prefer the main coordinate axes, i.e., have the standardisation dominated by them. I suspect you'd get something very similar to Euclidean (with standardised variables) from that. (I'll think about my word use of "preferring directions" and how Euclidean does that - I see your point but I mean something somewhat different; it's hard to put the intuition in words.) May 16, 2023 at 15:49
• But my point is that a spherical penalty, by definition, does not prefer any axes. So I suppose I am not correctly interpreting what you mean by some of these terms, but I truly cannot guess what you do mean.
– whuber
May 16, 2023 at 15:51
• @whuber OK, I reworded this, hopefully it says more clearly what I wanted to say, and in particular what I wrote was about Euclidean distance with standardisation (as Mahalanobis does this implicitly as well, but "in all directions" rather than only along the main coordinate axes). May 16, 2023 at 16:04