Mahalanobis distance between two bivariate distributions with different covariances

Question

The question is pretty much contained in the title. What is the Mahalanobis distance for two distributions of different covariance matrices? What I have found till now assumes the same covariance for both distributions, i.e., something of this sort:

$$\Delta^T \Sigma^{-1} \Delta$$

What if I have two different $\Sigma$s?

Note:- The problem is this: there are two bivariate distributions that have the same dimensions but that are rotated and translated with respect to each other (sorry I come from a pure mathematical background, not a statistics one). I need to measure their degree of overlap/distance.

*Update: * What might or might not be implicit in what I'm asking is that I need a distance between the means of the two distributions. I know where the means are, but since the two distributions are rotated with respect to one another, I need to assign different weights to different orientations and therefore a simple Euclidean distance between the means does not work. Now, as I have understood it, the Mahalanobis distance cannot be used to measure this information if the distributions are differently shaped (apparently it works with two multivariate normal distributions of identical covariances, but not in the general case). Is there a good measure that encodes this wish to encode orientations with different weights?

Answer 1

There are many notions of distance between probability distributions. Which one to use depends on your goals. Total variation distance is a natural way of measuring overlap between distributions. If you are working with multivariate Normals, the Kullback-Leibler Divergence is mathematically convenient. Though it is not actually a distance (as it fails to be symmetric and fails to obey the triangle inequality), it upper bounds the total variation distance — see Pinsker’s Inequality.

Answer 2

Intro As @vqv mentionned Total variation and Kullback Leibler are two interesting distance. The first one is meaningfull because it can be directly related to first and second type errors in hypothesis testing. The problem with the Total variation distance is that it can be difficult to compute. The Kullback Leibler distance is easier to compute and I will come to that later. It is not symetric but can be made symetric (somehow a little bit artificially).

Answer Something I mention here is that if $\mathcal{L}$ is the log likelihood ratio between your two gaussian measures $P_0,P_1$ (say that for $i=0,1$ $P_i$ has mean $\mu_i$ and covariance $C_i$) error measure that is also interseting (in the gaussian case I found it quite central actually) is

$$ \|\mathcal{L}\|^2_{L_2(P_{1/2})} $$

for a well chosen $P_{1/2}$.

In simple words:

• there might be different interesting "directions" rotations, that are obtained using your formula with one of the "interpolated" covariance matrices $\Sigma=C_{i,1/2}$ ($i=1,2,3,4$ or $5$) defined at the end of this post (the number $5$ is the one you propose in your comment to your question).
• since your two distributions have different covariances, it is not sufficiant to compare the means, you also need to compare the covariances.

Let me explain you why this is my feeling, how you can compute this in the case of $C_1\neq C_0$ and how to choose $P_{1/2}$.

Linear case If $C_1=C_0=\Sigma$.

$$\sigma= \Delta \Sigma^{-1} \Delta=\|2\mathcal{L}\|^2_{L_2(P_{1/2})}$$

where $P_{1/2}$ is the "interpolate" between $P_1$ and $P_0$ (gaussian with covariance $\Sigma$ and mean $(\mu_1+\mu_0)/2$). Note that in this case, the Hellinger distance, the total variation distance can all be written using $\sigma$.

How to compute $\mathcal{L}$ in the general case A natural question that arises from your question (and mine) is what is a natural "interpolate" between $P_1$ and $P_0$ when $C_1\neq C_0$. Here the word natural may be user specific but for example it may be related to the best interpolation to have a tight upper bound with another distance (e.g. $L_1$ distance here)

Writting $$ \mathcal{L}= \phi (C^{-1/2}_i(x-\mu_i))-\phi (C^{-1/2}_j(x-\mu_j))-\frac{1}{2}\log \left ( C_iC_j^{-}\right ) $$ ($i=0,j=1$) may help to see where is the interpolation task, but :

$$\mathcal{L}(x)=-\frac{1}{2}\langle A_{ij}(x-s_{ij}),x-s_{ij}\rangle_{\mathbb{R}^p}+\langle G_{ij},x-s_{ij}\rangle_{\mathbb{R}^p}-c_{ij}, \;[1]$$

with

$$A_{ij}=C_i^{-}-C_j^{-},\;\; G_{ij}=S_{ij}m_{ij},\;\; S_{ij}=\frac{C_i^{-}+C_j^{-}}{2}, $$ $$ c_{ij}=\frac{1}{8}\langle A_{ij} m_{ij},m_{ij}\rangle_{\mathbb{R}^p}+\frac{1}{2}\log|\det(C_j^{-}C_i)| $$

and

$$ m_{ij}=\mu_i-\mu_j \;\; and\;\; s_{ij}=\frac{\mu_i+\mu_j}{2}$$

is more relevant for computational purpose. For any gaussian $P_{1/2}$ with mean $s_{01}$ and covariance $C$ the calculation of $\|\mathcal{L}\|^2_{L_2(P_{1/2})}$ from Equation $1$ is a bit technical but faisible. You might also use it to compute the Kulback leibler distance.

What interpolation should we choose (i.e. how to choose $P_{1/2}$) It is clearly understood from Equation $1$ that there are many different candidates for $P_{1/2}$ (interpolate) in the "quadratic" case. The two candidates I found "most natural" (subjective:) ) arise from defining for $t\in [0,1]$ a gaussian distribution $P_t$ with mean $t\mu_1+(1-t)\mu_0$:

1. $P^1_t$ as the distribution of $$ \xi_t=t\xi_1+(1-t)\xi_0$$ (where $\xi_i$ is drawn from $P_i$ $i=0,1$) which has covariance $C_{t,1}=(tC_1^{1/2}+(1-t)C_0^{1/2})^2$).
2. $P^2_t$ with inverse covariance $C_{t,2}^{-1}=tC_{1}^{-1}+(1-t)C_0^{-1}$
3. $P^3_t$ with covariance $C_{t,3}=tC_1+(1-t)C_0$
4. $P^4_t$ with inverse covariance $C_{t,4}^{-1}=(tC^{-1/2}_1+(1-t)C^{-1/2}_0)^{2}$

EDIT: The one you propose in a comment to your question could be $C_{t,5}=C_1^{t}C_0^{1-t}$, why not ...

I have my favorite choice which is not the first one :) don't have much time to discuss that here. Maybe I'll edit this answer later...

Answer 3

This is old, but for others who are reading this, the covariance matrix reflects the rotation of the gaussian distributions and the mean reflects the translation or central position of the distribution. To evaluate the mahab distance, it is simply D= ( (m2-m1) * inv( (C1 + C2)/2 ) * (m2-m1)'). Now if you suspect that the two bivariate distributions are the same, but you suspect that they have been rotated, then compute the two pairs of eigenvectors and eigenvalues for each distribution. The eigenvectors point in the direction of the spread of the bivariate data along the major and minor axes and the eigenvalues denote the length of this spread. If the eigenvalues are same, then the two distributions are the same but rotated. Take acos of the dot product between the eigenvectors to get the angle of rotation.