16
$\begingroup$

I have seen formulas on Wikipedia. that relate Mahalanobis distance and Leverage:

Mahalanobis distance is closely related to the leverage statistic, $h$, but has a different scale: $$D^2 = (N - 1)(h - \tfrac{1}{N}).$$

In a linked article, Wikipedia describes $h$ in these terms:

In the linear regression model, the leverage score for the $i^{th}$ data unit is defined as:$$h_{ii}=(H)_{ii},$$ the $i^{th}$ diagonal element of the hat matrix $H=X(X^{\top}X)^{-1}X^{\top}$, where $^{\top}$ denotes the matrix transpose.

I can't find a proof anywhere. I tried to start from the definitions but I can't make any progress. Anyone can give some hint?

$\endgroup$

3 Answers 3

13
$\begingroup$

My description of the Mahalanobis distance at Bottom to top explanation of the Mahalanobis distance? includes two key results:

  1. By definition, it does not change when the regressors are uniformly shifted.

  2. The squared Mahalanobis distance between vectors $x$ and $y$ is given by $$D^2(x,y) = (x-y)^\prime \Sigma^{-1}(x-y)$$ where $\Sigma$ is the covariance of the data.

(1) allows us to assume the means of the regressors are all zero. It remains to compute $h_i$. However, for the claim to be true, we need to add one more assumption:

The model must include an intercept.

Allowing for this, let there be $k \ge 0$ regressors and $n$ data, writing the value of regressor $j$ for observation $i$ as $x_{ij}$. Let the column vector of these $n$ values for regressor $j$ be written $\mathbf{x}_{,j}$ and the row vector of these $k$ values for observation $i$ be written $\mathbf{x}_i$. Then the model matrix is

$$X = \pmatrix{ 1 &x_{11} &\cdots &x_{1k} \\ 1 &x_{21} &\cdots &x_{2k} \\ \vdots &\vdots &\vdots &\vdots \\ 1 &x_{n1} &\cdots &x_{nk}}$$

and, by definition, the hat matrix is

$$H = X(X^\prime X)^{-1} X^\prime,$$

whence entry $i$ along the diagonal is

$$h_i = h_{ii} = (1; \mathbf{x}_i) (X^\prime X)^{-1} (1; \mathbf{x}_i)^\prime.\tag{1}$$

There's nothing for it but to work out that central matrix inverse--but by virtue of the first key result, it's easy, especially when we write it in block-matrix form:

$$X^\prime X = n\pmatrix{1 & \mathbf{0}^\prime \\ \mathbf{0} & C}$$

where $\mathbf{0} = (0,0,\ldots,0)^\prime$ and

$$C_{jk} = \frac{1}{n} \sum_{i=1}^n x_{ij} x_{ik} = \frac{n-1}{n}\operatorname{Cov}(\mathbf{x}_j, \mathbf{x}_k) = \frac{n-1}{n}\Sigma_{jk}.$$

(I have written $\Sigma$ for the sample covariance matrix of the regressors.) Because this is block diagonal, its inverse can be found simply by inverting the blocks:

$$(X^\prime X)^{-1} = \frac{1}{n}\pmatrix{1 & \mathbf{0}^\prime \\ \mathbf{0} & C^{-1}} = \pmatrix{\frac{1}{n} & \mathbf{0}^\prime \\ \mathbf{0} & \frac{1}{n-1}\Sigma^{-1}}.$$

From the definition $(1)$ we obtain

$$\eqalign{ h_i &= (1; \mathbf{x}_i) \pmatrix{\frac{1}{n} & \mathbf{0}^\prime \\ \mathbf{0} & \frac{1}{n-1}\Sigma^{-1}}(1; \mathbf{x}_i)^\prime \\ &=\frac{1}{n} + \frac{1}{n-1}\mathbf{x}_i \Sigma^{-1}\mathbf{x}_i^\prime \\ &=\frac{1}{n} + \frac{1}{n-1} D^2(\mathbf{x}_i, \mathbf{0}). }$$

Solving for the squared Mahalanobis length $D_i^2 = D^2(\mathbf{x}_i, \mathbf{0})$ yields

$$D_i^2 = (n-1)\left(h_i - \frac{1}{n}\right),$$

QED.

Looking back, we may trace the additive term $1/n$ to the presence of an intercept, which introduced the column of ones into the model matrix $X$. The multiplicative term $n-1$ appeared after assuming the Mahalanobis distance would be computed using the sample covariance estimate (which divides the sums of squares and products by $n-1$) rather than the covariance matrix of the data (which divides the sum of squares and products by $n$).


The chief value of this analysis is to impart a geometric interpretation to the leverage, which measures how much a unit change in the response at observation $i$ will change the fitted value at that observation: high-leverage observations are at large Mahalanobis distances from the centroid of the regressors, exactly as a mechanically efficient lever operates at a large distance from its fulcrum.


R code to show that the relation indeed holds:

x <- mtcars

# Compute Mahalanobis distances
h <- hat(x, intercept = TRUE); names(h) <- rownames(mtcars)
M <- mahalanobis(x, colMeans(x), cov(x))

# Compute D^2 of the question
n <- nrow(x); D2 <- (n-1)*(h - 1/n)

# Compare.
all.equal(M, D2)               # TRUE
print(signif(cbind(M, D2), 3))
$\endgroup$
12
  • 1
    $\begingroup$ Thanks for the post @whuber ! For sanity check, here is R code to show that the relation indeed holds: x <- mtcars rownames(x) <- NULL colnames(x) <- NULL n <- nrow(x) h <- hat(x, T) mahalanobis(x, colMeans(x), cov(x)) (n-1)*(h - 1/n) all.equal(mahalanobis(x, colMeans(x), cov(x)), (n-1)*(h - 1/n)) $\endgroup$
    – Tal Galili
    Jul 16, 2019 at 7:21
  • 1
    $\begingroup$ @Tal I didn't think I needed a sanity check--but thank you for the code. :-) I have made modifications to clarify it and its output a little. $\endgroup$
    – whuber
    Jul 16, 2019 at 12:33
  • 1
    $\begingroup$ @whuber, I wanted an example that shows how to make the equality works (making clear to me that I got the assumptions right). I've also extended the relevant Wiki entry: en.wikipedia.org/wiki/… (feel free to also expend on it there, as you see fit :) ) $\endgroup$
    – Tal Galili
    Jul 16, 2019 at 13:33
  • 1
    $\begingroup$ How do we justify that $X^T X$ is the sample covariance rather than the sample second moment? I buy that the Mahalanobis distance does not change when the regressors are all shifted by the same amount, but why does that mean we can assume the data is already centered when computing $X^T X$? @whuber $\endgroup$ Dec 14, 2021 at 19:24
  • 1
    $\begingroup$ See my above comment. My concern is before the Mahalanobis distance appears. You reach $(x_i - \mu)^T \Sigma^{-1} (x_i - \mu)$ and then say this is the squared Mahalanobis distance, which is correct. I'm pointing out that $(x_i - \mu)^T \Sigma^{-1} (x_i - \mu)$ is reached only if the data is centered; if we don't make that assumption, we instead reach $x_i^T (\Sigma + \mu \mu^T)^{-1} x_i$. How do we get the Mahalanobis distance from that? $\endgroup$ Dec 14, 2021 at 20:08
0
$\begingroup$

Here is an attempt to bridge the gap between $\textbf{X}^T\textbf{X}$ and the sample covariance matrix via block matrix inverse. Note that I have defined $\hat\Sigma$ with $1/n$. Replacing it by $1/(n-1)$ gives the form in Wiki. $$ \textbf{X}=\begin{pmatrix} 1 & x_{11} & \cdots & x_{1 p} \\ 1 & x_{21} & \cdots & x_{2 p} \\ \vdots & \vdots & \vdots & \vdots \\ 1 & x_{n 1} & \cdots & x_{n p} \end{pmatrix} $$

The hat matrix is $$ \textbf{H}=\textbf{X}\left(\textbf{X}^T \textbf{X}\right)^{-1} \textbf{X}^T, $$ with its $i$th diagonal element $$h_{ii}= \begin{pmatrix} 1 \\ \mathbf{x}_i \end{pmatrix}^T \left(\textbf{X}^T \textbf{X}\right)^{-1} \begin{pmatrix} 1 \\ \mathbf{x}_i \end{pmatrix}.$$

We have $$ \textbf{X}^T\textbf{X} = \begin{pmatrix} n & \sum_{i=1}^nx_{i1} & \cdots & \sum_{i=1}^nx_{ip} \\ \sum_{i=1}^nx_{i1} & \sum_{i=1}^nx_{i1}^2 & \cdots & \sum_{i=1}^nx_{i1}x_{ip} \\ \vdots & \vdots & \vdots & \vdots \\ \sum_{i=1}^nx_{ip} & \sum_{i=1}^nx_{i1}x_{ip} & \cdots & \sum_{i=1}^nx_{ip}^2 \end{pmatrix}. $$ By the formula of block matrix inverse, $$ (\textbf{X}^T\textbf{X})^{-1}= \frac{1}{n}\begin{pmatrix} 1+ \overline{\textbf{x}}^T\hat\Sigma^{-1}\overline{\textbf{x}} & -\overline{\textbf{x}}^T\hat\Sigma^{-1}\\ -\hat\Sigma^{-1}\overline{\textbf{x}}& \hat\Sigma^{-1} \end{pmatrix}, $$ where $$ \hat\Sigma\triangleq \frac{1}{n}\begin{pmatrix} \sum_{i=1}^nx_{i1}^2 -n^{-1}(\sum_{i=1}^nx_{i1})^2 & \cdots & \sum_{i=1}^nx_{i1}x_{ip} - n^{-1}\sum_{i=1}^nx_{i1}\sum_{i=1}^nx_{ip} \\ \vdots & \vdots & \vdots \\ \sum_{i=1}^nx_{i1}x_{ip} -n^{-1}\sum_{i=1}^nx_{i1}\sum_{i=1}^nx_{ip}& \cdots & \sum_{i=1}^nx_{ip}^2-n^{-1}(\sum_{i=1}^nx_{ip})^2 \end{pmatrix}. $$ Therefore, \begin{align*} h_{ii}=&\ \frac{1}{n}\begin{pmatrix}1+ \overline{\textbf{x}}^T\hat\Sigma^{-1}\overline{\textbf{x}} - \textbf{x}_i^T\hat\Sigma^{-1}\overline{\textbf{x}}, -\overline{\textbf{x}}^T\hat\Sigma^{-1} + \textbf{x}_i^T \hat\Sigma^{-1} \end{pmatrix} \begin{pmatrix} 1 \\ \mathbf{x}_i \end{pmatrix}\\ =&\ \frac{1}{n}(1+ \overline{\textbf{x}}^T\hat\Sigma^{-1}\overline{\textbf{x}} -2 \textbf{x}_i^T\hat\Sigma^{-1}\overline{\textbf{x}} + \textbf{x}_i \hat\Sigma^{-1} \textbf{x}_i)\\ =& \frac{1}{n}+\frac{1}{n}(\textbf{x}_i-\overline{\textbf{x}})^T\hat\Sigma^{-1}(\textbf{x}_i-\overline{\textbf{x}}). \end{align*}

$\endgroup$
-1
$\begingroup$

I fear @whuber's answer assumes a key step that is not assumable. I don't know how to fix it, but I want to explain the mistake. The problem is the statement "(1) allows us to assume the means of the regressors are all zero." While that is true for the Mahalanobis distance, @whuber assumes the raw data to be mean zero before any Malanobis distance enters the picture. I don't know how this is justifiable.

To see this, let's redo the calculation without assuming that $X$ is centered. As above, we have:

$$[H]_{ii} = \begin{bmatrix} 1 \\ x_i \end{bmatrix}^T (X^T X)^{-1} \begin{bmatrix} 1 \\ x_i \end{bmatrix}$$

One can see that $X^T X = \sum_{i} x_i x_i^T$. @whuber assumes that the data is centered and thus writes:

$$X^T X = n \begin{bmatrix} 1 & 0^T \\ 0 & C \end{bmatrix}$$

But how is this justified? No Mahalanobis distance has (yet) entered the picture. Nothing tells us that the data is centered. In actuality, the correct expression is:

$$X^T X = n \begin{bmatrix} 1 & \mu^T \\ \mu & \Sigma + \mu \mu^T \end{bmatrix}$$

An easy block inversion is no longer possible and I don't know how to complete the derivation.

$\endgroup$
5
  • $\begingroup$ -1 Please read my answer again: I make no such assumptions. If you wish to start with your last expression, feel free to do so: every occurrence of "$\mu$" will vanish after the initial row and column reduction steps. Thus, with extra initial work, you will arrive at the point where I began and you can continue from there. If this isn't perfectly clear, carry out a calculation by hand with a tiny design matrix, such as a $3\times 2$ matrix. $\endgroup$
    – whuber
    Dec 15, 2021 at 2:36
  • $\begingroup$ Whether you call it an assumption or not, you do not show that $\mu$ vanishes. You just say that it does. Even if it does, the onus falls on you to show that it does. $\endgroup$ Dec 15, 2021 at 2:46
  • $\begingroup$ @whuber , you're a bully. I'm correct in pointing out that your answer is incomplete without showing that "every occurrence of "𝜇" will vanish after the initial row and column reduction steps" is true and you're downvoting me $\endgroup$ Dec 15, 2021 at 2:54
  • $\begingroup$ We have had an extensive discussion about this. Trying to address your misunderstanding by posting an incorrect answer is counterproductive because it risks confusing others. My downvote is an effort to signal that there are fundamental problems in your answer. I am sorry if you view that as "bullying:" I see such commenting and voting as being a necessary part of how this site helps develop high-quality answers. For the same reasons, your revenge downvote of my answer is unrelated to its usefulness and thereby detracts from the quality of this site. $\endgroup$
    – whuber
    Dec 15, 2021 at 13:52
  • 2
    $\begingroup$ Rylan Schaeffer -- I shared your concern about mean-shifting the regressor observations. It made sense that it wouldn't affect $D^2$, but I was confused how it wouldn't affect $H$. It turns out that H is invariant to constant shifts in the the regressing variables -- see the discussion on this on the mathematics stack exchange: math.stackexchange.com/questions/4127584/…. Because of this, we can follow @whuber's work simply noting that not only $D^2$ but also $H$ remain unchanged after the mean shift. $\endgroup$ Nov 3, 2023 at 19:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.