Bottom to top explanation of the Mahalanobis distance?

Question

I'm studying pattern recognition and statistics and almost every book I open on the subject I bump into the concept of Mahalanobis distance. The books give sort of intuitive explanations, but still not good enough ones for me to actually really understand what is going on. If someone would ask me "What is the Mahalanobis distance?" I could only answer: "It's this nice thing, which measures distance of some kind" :)

The definitions usually also contain eigenvectors and eigenvalues, which I have a little trouble connecting to the Mahalanobis distance. I understand the definition of eigenvectors and eigenvalues, but how are they related to the Mahalanobis distance? Does it have something to do with changing the base in Linear Algebra etc.?

I have also read these former questions on the subject:

• What is Mahalanobis distance, & how is it used in pattern recognition?

• Intuitive explanations for Gaussian distribution function and mahalanobis distance (Math.SE)

I have also read this explanation.

The answers are good and pictures nice, but still I don't really get it...I have an idea but it's still in the dark. Can someone give a "How would you explain it to your grandma"-explanation so that I could finally wrap this up and never again wonder what the heck is a Mahalanobis distance? :) Where does it come from, what, why?

UPDATE:

Here is something which helps understanding the Mahalanobis formula:

https://math.stackexchange.com/questions/428064/distance-of-a-test-point-from-the-center-of-an-ellipsoid

Answer 1

Here is a scatterplot of some multivariate data (in two dimensions):

What can we make of it when the axes are left out?

Introduce coordinates that are suggested by the data themselves.

The origin will be at the centroid of the points (the point of their averages). The first coordinate axis (blue in the next figure) will extend along the "spine" of the points, which (by definition) is any direction in which the variance is the greatest. The second coordinate axis (red in the figure) will extend perpendicularly to the first one. (In more than two dimensions, it will be chosen in that perpendicular direction in which the variance is as large as possible, and so on.)

We need a scale. The standard deviation along each axis will do nicely to establish the units along the axes. Remember the 68-95-99.7 rule: about two-thirds (68%) of the points should be within one unit of the origin (along the axis); about 95% should be within two units. That makes it easy to eyeball the correct units. For reference, this figure includes the unit circle in these units:

That doesn't really look like a circle, does it? That's because this picture is distorted (as evidenced by the different spacings among the numbers on the two axes). Let's redraw it with the axes in their proper orientations--left to right and bottom to top--and with a unit aspect ratio so that one unit horizontally really does equal one unit vertically:

You measure the Mahalanobis distance in this picture rather than in the original.

What happened here? We let the data tell us how to construct a coordinate system for making measurements in the scatterplot. That's all it is. Although we had a few choices to make along the way (we could always reverse either or both axes; and in rare situations the directions along the "spines"--the principal directions--are not unique), they do not change the distances in the final plot.

---

Technical comments

(Not for grandma, who probably started to lose interest as soon as numbers reappeared on the plots, but to address the remaining questions that were posed.)

• Unit vectors along the new axes are the eigenvectors (of either the covariance matrix or its inverse).

• We noted that undistorting the ellipse to make a circle divides the distance along each eigenvector by the standard deviation: the square root of the covariance. Letting $C$ stand for the covariance function, the new (Mahalanobis) distance between two points $x$ and $y$ is the distance from $x$ to $y$ divided by the square root of $C(x-y, x-y)$. The corresponding algebraic operations, thinking now of $C$ in terms of its representation as a matrix and $x$ and $y$ in terms of their representations as vectors, are written $\sqrt{(x-y)'C^{-1}(x-y)}$. This works regardless of what basis is used to represent vectors and matrices. In particular, this is the correct formula for the Mahalanobis distance in the original coordinates.

• The amounts by which the axes are expanded in the last step are the (square roots of the) eigenvalues of the inverse covariance matrix. Equivalently, the axes are shrunk by the (roots of the) eigenvalues of the covariance matrix. Thus, the more the scatter, the more the shrinking needed to convert that ellipse into a circle.

• Although this procedure always works with any dataset, it looks this nice (the classical football-shaped cloud) for data that are approximately multivariate Normal. In other cases, the point of averages might not be a good representation of the center of the data or the "spines" (general trends in the data) will not be identified accurately using variance as a measure of spread.

• The shifting of the coordinate origin, rotation, and expansion of the axes collectively form an affine transformation. Apart from that initial shift, this is a change of basis from the original one (using unit vectors pointing in the positive coordinate directions) to the new one (using a choice of unit eigenvectors).

• There is a strong connection with Principal Components Analysis (PCA). That alone goes a long way towards explaining the "where does it come from" and "why" questions--if you weren't already convinced by the elegance and utility of letting the data determine the coordinates you use to describe them and measure their differences.

• For multivariate Normal distributions (where we can carry out the same construction using properties of the probability density instead of the analogous properties of the point cloud), the Mahalanobis distance (to the new origin) appears in place of the "$x$" in the expression $\exp(-\frac{1}{2} x^2)$ that characterizes the probability density of the standard Normal distribution. Thus, in the new coordinates, a multivariate Normal distribution looks standard Normal when projected onto any line through the origin. In particular, it is standard Normal in each of the new coordinates. From this point of view, the only substantial sense in which multivariate Normal distributions differ among one another is in terms of how many dimensions they use. (Note that this number of dimensions may be, and sometimes is, less than the nominal number of dimensions.)

Answer 2

My grandma cooks. Yours might too. Cooking is a delicious way to teach statistics.

Pumpkin Habanero cookies are awesome! Think about how wonderful cinnamon and ginger can be in Christmas treats, then realize how hot they are on their own.

The ingredients are:

3/4 c Pumpkin, canned
3/4 c Sugar, brown, light
1/2 c Sour cream
1 Eggs; beaten
1 ts Vanilla
1 c Raisins, seedless
1/2 c Walnuts; chopped
1 1/2 c Flour, all-purpose
1 ts Cinnamon
1/2 ts Ginger
1/2 ts Baking soda
1/2 ts Salt
1/2 ts Chiles, habanero, ground
1/4 ts Allspice
1/4 ts Nutmeg

Imagine your coordinate axes for your domain being the ingredient volumes. Sugar. Flour. Salt. Baking Soda. Variation along those directions, all else being equal, doesn't have nearly the impact to the flavor quality as variation in count of habanero peppers. A 10% change in flour or butter is going to make it less great, but not killer. Adding just a small amount more habanero will knock you over a flavor cliff from addictive-dessert to testosterone based pain-contest.

Mahalanobis isn't as much a distance in "ingredient volumes" as it is distance away from "best taste". The really "potent" ingredients, ones very sensitive to variation, are the ones you must most carefully control.

If you think about any Gaussian distribution vs. the Standard Normal distribution, what is the difference? Center and scale based on central tendency (mean) and variation tendency (standard deviation). One is the coordinate transform of the other. Mahalanobis is that transform. It shows you what the world looks like if your distribution of interest was re-cast as a standard normal instead of a Gaussian.

Answer 3

I'd like to add a little technical information to Whuber's excellent answer. This information might not interest grandma, but perhaps her grandchild would find it helpful. The following is a bottom-to-top explanation of the relevant linear algebra.

Mahalanobis distance is defined as $d(x,y)=\sqrt{(x-y)^T\Sigma^{-1}(x-y)}$, where $\Sigma$ is an estimate of the covariance matrix for some data; this implies it is symmetric. If the columns used to estimate $\Sigma$ are not linearly dependent, $\Sigma$ is positive definite. Symmetric matrices are diagonalizable and their eigenvalues and eigenvectors are real. PD matrices have eigenvalues which are all positive. The eigenvectors can be chosen to have unit length, and are orthogonal (i.e. orthonormal) so we can write $\Sigma=Q^TDQ$ and $\Sigma^{-1}=QD^{-\frac{1}{2}}D^{-\frac{1}{2}}Q^T$.

Plugging that into the distance definition, $$\begin{align} d(x,y) &= \sqrt{\left[(x-y)^TQ\right]D^{-\frac{1}{2}}D^{-\frac{1}{2}}\left[Q^T(x-y)\right]} \\ &=\sqrt{z^Tz} \end{align} $$

• The products in the square brackets are transposes, and the effect of multiplication by $Q$ is rotating the vector $(x-y)$ into an orthogonal basis.
• $D^{-\frac{1}{2}}$ is diagonal, precisely the inverse standard deviation of each feature in the orthogonal space.${}^*$ That is, $D^{-1}$ a precision matrix (inverse covariance). The matrix is diagonal because the data are in an orthogonal basis. The effect is to transform what Whuber calls a rotated ellipse into a circle by "flattening" its axes.
• Clearly $z^Tz$ is measured in the squared units, so taking the square root returns the distance into the original units.

---

${}^*$ We have data $X$ with observations stored in $n$ rows and features in $p$ columns and all column means are 0. Rotating it into an orthogonal basis is done via $\tilde X = XQ$. The estimator of the covariance matrix for the orthogonal data $\tilde X$ is given by $$ \begin{align} \tilde \Sigma &= \frac{1}{n-1} \tilde X^T \tilde X \\ &= \frac{1}{n-1} Q^TX^TXQ \\ &= \frac{1}{n-1} Q^T (n-1)\Sigma Q \\ &= Q^T Q D Q^T Q \\ &= D \end{align}$$

Because $D$ is a covariance matrix, we know that the variances of each feature are on the diagonal. The square root of the variance is the standard deviation.

Answer 4

As a starting point, I would see the Mahalanobis distance as a suitable deformation of the usual Euclidean distance $d(x,y)=\sqrt{\langle x,y \rangle}$ between vectors $x$ and $y$ in $\mathbb R^{n}$. The extra piece of information here is that $x$ and $y$ are actually random vectors, i.e. 2 different realizations of a vector $X$ of random variables, lying in the background of our discussion. The question that the Mahalanobis tries to address is the following:

"how can I measure the "dissimilarity" between $x$ and $y$, knowing that they are realization of the same multivariate random variable?"

Clearly the dissimilarity of any realization $x$ with itself should be equal to 0; moreover, the dissimilarity should be a symmetric function of the realizations and should reflect the existence of a random process in the background. This last aspect is taken into consideration by introducing the covariance matrix $C$ of the multivariate random variable.

Collecting the above ideas we arrive quite naturally at

$$D(x,y)=\sqrt{(x-y)\,C^{-1}(x-y)} $$

If the components $X_i$ of the multivariate random variable $X=(X_1,\dots,X_n)$ are uncorrelated, with, for example $C_{ij}=\delta_{ij}$ (we "normalized" the $X_i$'s in order to have $Var(X_i)=1$), then the Mahalanobis distance $D(x,y)$ is the Euclidean distance between $x$ and $y$. In presence non trivial correlations, the (estimated) correlation matrix $C(x,y)$ "deforms" the Euclidean distance.

Answer 5

I'll try to explain you as simply as possible:

Mahalanobis distance measures the distance of a point x from a data distribution. The data distribution is characterized by a mean and the covariance matrix, thus is hypothesized as a multivariate gaussian.

It is used in pattern recognition as similarity measure between the pattern (data distribution of training example of a class) and the test example. The covariance matrix gives the shape of how data is distributed in the feature space.

The figure indicates three different classes and the red line indicates the same Mahalanobis distance for each class. All points lying on the red line have the same distance from the class mean, because it is used the covariance matrix.

The key feature is the use of covariance as a normalization factor.

Answer 6

Let's consider the two variables case. Seeing this picture of bivariate normal (thanks @whuber), you cannot simply claim that AB is larger than AC. There is a positive covariance; the two variables are related to each other.

You can apply simple Euclidean measurements (straight lines like AB and AC) only if the variables are

1. independent
2. have variances equal to 1.

Essentially, Mahalanobis distance measure does the following: it transforms the variables into uncorrelated variables with variances equal to 1, and then calculates simple Euclidean distance.

Answer 7

I might be a bit late for answering this question. This paper in The Mahalanobis distance by MaesschalckD and Et. al. is a good start for understanding the Mahalanobis distance. They've provided a complete example with numerical values. What I like about it is the geometric representation of the problem is presented.

Answer 8

Just to add to the excellent explanations above, the Mahalanobis distance arises naturally in (multivariate) linear regression. This is a simple consequence of some of the connections between the Mahalanobis distance and the Gaussian distribution discussed in the other answers, but I think it's worth spelling out anyway.

Suppose we have some data $(x_1, y_1), \ldots, (x_N, y_N)$, with $x_i \in \mathbb{R}^n$ and $y_i \in \mathbb{R}^m$. Let's assume that there exists a parameter vector $\beta_0 \in \mathbb{R}^m$ and a parameter matrix $\beta_1 \in \mathbb{R}^{m \times n}$ such that $y_i = \beta_0 + \beta_1 x_i + \epsilon_i$, where $\epsilon_1, \ldots, \epsilon_N$ are iid $m$-dimensional Gaussian random vectors with mean $0$ and covariance $C$ (and they are independent of the $x_i$). Then $y_i$ given $x_i$ is Gaussian with mean $\beta_0 + \beta_1 x_i$ and covariance $C$.

It follows that the negative log-likelihood of $y_i$ given $x_i$ (as a function of $\beta = (\beta_0, \beta_1)$) is given by \begin{equation} -\log p(y_i \mid x_i; \beta) = \frac{m}{2} \log (2\pi\det C) + \frac{1}{2} (y_i - (\beta_0 + \beta_1 x_i))^\top C^{-1} (y_i - (\beta_0 + \beta x_i)). \end{equation} We are taking the covariance $C$ to be constant, so \begin{equation} \operatorname{argmin}_\beta [-\log p(y_i \mid x_i; \beta)] = \operatorname{argmin}_\beta D_C(\beta_0 + \beta_1 x_i, y_i), \end{equation} where \begin{equation} D_C(\hat y, y) = \sqrt{(y - \hat y)^\top C^{-1} (y - \hat y)} \end{equation} is the Mahalanobis distance between $\hat y, y \in \mathbb{R}^m$.

By independence, the log-likelihood $\log p({\bf y} \mid {\bf x}; \beta)$ of ${\bf y} = (y_1, \ldots, y_N)$ given ${\bf x} = (x_1, \ldots, x_N)$ is given by the sum \begin{equation} \log p({\bf y} \mid {\bf x}; \beta) = \sum_{i=1}^N \log p(y_i \mid x_i; \beta) \end{equation} Therefore, \begin{equation} \operatorname{argmin}_\beta [-\log p({\bf y} \mid {\bf x}; \beta)] = \operatorname{argmin}_\beta \frac{1}{N} \sum_{i=1}^N D_C(\beta_0 + \beta_1 x_i, y_i), \end{equation} where the factor $1/N$ does not affect the argmin.

In summary, the coefficients $\beta_0, \beta_1$ that minimize the negative log-likelihood (i.e. maximize the likelihood) of the observed data also minimize the empirical risk of the data with loss function given by the Mahalanobis distance.

Answer 9

Mahalanobis distance is an euclidian distance (natural distance) wich take into account the covariance of data. It give a bigger weight to noisy component and so is very usefull to check for similarity between two datasets.

As you can see in your exemple here when variables are correlated, the distribution is shifted into one direction. You may want to remove this effects. If you take correlation into account in your distance, you can remove the shift effect.

Answer 10

Standardise distance to the mean of normal distribution

In my understanding, Z-score and Mahalanobis distance are the standardising methods to measure the closeness to the mean of a distribution, where we need to consider the direction and variance of the dispersions.

Question

Which one, A or B, is closer to the mean of the distribution, or conversely which is more remote outlier to the distribution.

Direction and variance of the dispersions

To answer the question, I need to consider the direction and variance of the dispersions in the distribution.

Although the visible distances to A and B look similar, B belongs to the distribution and A is an outlier. The distance to the mean should be shorter (closer to the mean) along the direction to B with the larger dispersion (higher variance).

(Direction, Variance) to (Eigenvector, Eigenvalue)

The eigenvectors $u_i$ tell the directions of the dispersions, and the eigenvalues $\lambda_i$ tell the variances of the dispersions. Higher eigenvalue means larger variance. Then if I scale the space by $\frac {1}{\sqrt{\lambda_A}}$ in the A direction and $\frac {1}{\sqrt{\lambda_B}}$ in the B direction, I can decide which is close to the mean by comparing:

$$\frac {\Vert A-\mu \Vert}{\sqrt{\lambda_A}} \text {and} \frac {\Vert B-\mu \Vert}{\sqrt{\lambda_B}}$$

The distance in the direction with higher variance gets shorter.

---

Standardise distance to mean

In a univariate normal distribution where there is only one direction of dispersion and one variance $\sigma^2$, I can standardise the distance via Z-score as in the top half of the snapshot.

In multivariate normal distribution, I can standardise the distance as in the bottom of the snapshot via the steps:

1. Project $(X - \mu)$ into the U space where the unit eigenvectors $u_i$ are the orthonormal axes. This operation is changing the coordinate in X space to the coordinate in U space.
2. Scale to the $u_i$ directions by $$\frac {1}{\sqrt{\lambda_i}}$$ where $\lambda_i$ is the eigenvalues for the eigenvectors $u_i$ and $\lambda_i$ is sorted by descending order so that $\lambda_1 \ge \lambda_2$.

• Division by the variance $\sigma^2$ in univariate corresponds to inverse of covariance matrix $\Sigma^{-1}$ in multivariate.

• Scale by $\frac {1}{\sigma}$ in univariate corresponds to scale by $\frac {1}{\sqrt{\lambda}}$ in multivariate.

Inverse Cholesky transformation

The steps of projecting into U space and scaling are, in my understanding, Inverse Cholesky Transformation as explained in Use the Cholesky transformation to correlate and uncorrelate variables.

Eigen Decomposition

Covariance matrix of the distribution X, which is $\Sigma$, can be decomposed as $\Sigma = U \Lambda U^T$ where $U$ is a matrix whose column is an eigenvector $u_i$ and $\Lambda$ is the eigenvalue matrix (sorted descending order).

The first step of projecting into U space is applying the matrix $U^T$. This is the operation of de-correlation or acquiring the principal components in PCA (when eigen decomposition is being used instead of SVD).

By de-correlation, the multi variate distribution can be the product of independent univariate distributions.

Pattern Recognition and Machine Learning (Christopher Bishop) - 2.3 The Gaussian Distribution

---

Related Resources

• What is Mahalanobis distance?
• Mahalanobis Distance
• 5 Multivariate Normal Distribution (slides not the transcript)

• Is Mahalanobis distance equivalent to the Euclidean one on the PCA-rotated data?
• STAT 505 Applied Multivariate Statistical Analysis - 4.6 - Geometry of the Multivariate Normal Distribution
• (PP 6.7) Geometric intuition for the multivariate Gaussian (part 2)

• Deriving the formula for multivariate Gaussian distribution
• Pattern Recognition and Machine Learning (Christopher Bishop) - 2.3 The Gaussian Distribution

• Why does univariate Mahalanobis distance not match z-score?