I have read several times across different sources now that the definition of Cook's Distance, which is
$$D_i=\frac{\sum_{j=1}^n(\hat{y}_j-\hat{y}_{j(i)})^2}{ps^2}$$
(where $\hat{y}_j$ is the jth fitted response value, $\hat{y}_{j(i)}$ the jth fitted response value, where the fit does not include observation i, $p$ the number of coefficients in the model, $s^2$ the MSE)
is algebraically equivalent to the expression
$$\frac{r_i^2}{ps^2}\frac{h_{ii}}{(1-h_{ii})^2}$$
For $r_i$ the ith residual, $h_{ii}$ the ith leverage value.
Despite numerous sources stating this (lecture notes from different universities, wikipedia, Matlab's website, etc.), I have not been able to find a single proof of this. I would very much appreciate it if someone could direct me to one such proof. It seems like a completely non-trivial equivalence to me, and I am unsure why it's so difficult to find a proof of this.
2 Answers
I have found a source written by Cook himself (here is a pdf link I found):
Cook, R.D., (1977). Detection of Influential Observation in Linear Regression. Technometrics, 19(1), 15-18.
The derivation ends at equation 7 in that paper.
-
$\begingroup$ Thanks. I am curious, how did you find it? I've been looking for a while without finding this. $\endgroup$ Commented Dec 14, 2019 at 9:40
-
$\begingroup$ It was a long journey through the internet that I can't retrace, but someone mentioned a paper by Cook in 1979 as a reference for Cook's distance. In that paper, Cook references this 1977 paper as the introduction of (what we now call) Cook's distance. By luck he also derives the algebraic equivalent there. $\endgroup$ Commented Dec 15, 2019 at 0:03
I've read the paper and here is a summary of the proof : First we show that $$\sum_{j=1}^{n}(\hat{y}_{j}-\hat{y}_{j(i)})^2 = {(\hat{\beta}_{(i)}-\hat{\beta})'X'X(\hat{\beta}_{(i)}-\hat{\beta})}$$ Rewrite the right hand side: $${(\hat{\beta}_{(i)}-\hat{\beta})'X'X(\hat{\beta}_{(i)}-\hat{\beta})}={(X\hat{\beta}_{(i)}-X\hat{\beta})'(X\hat{\beta}_{(i)}-X\hat{\beta})} ={(\hat{Y}_{(i)}-\hat{Y})'I(\hat{Y}_{(i)}-\hat{Y})}$$ which is a quadratic form of $\sum_{j=1}^{n}(\hat{y}_{j}-\hat{y}_{j(i)})^2$. Therefore $D_{i}=\frac{(\hat{\beta}_{(i)}-\hat{\beta})'X'X(\hat{\beta}_{(i)}-\hat{\beta})}{ps^2}$. Now we need to find something to replace ${(\hat{\beta}_{(i)}-\hat{\beta})}$. We can partition $X'X$ into two parts: $$X'X = (X_{(i)}'X_{(i)})+x_{i}x_{i}'$$ where $x_{i}'$ is the ith row of X. This is also possible for $X'Y$ : $$X'Y = (X_{(i)}'Y_{(i)})+x_{i}y_{i}$$ Let's find a formula for $(X_{(i)}'X_{(i)})^{-1}$. Use Sherman–Morrison formula to derive the following formula: $$(X_{(i)}'X_{(i)})^{-1}=(X'X - x_{i}x_{i}')^{-1} = (X'X)^{-1} + \frac{(X'X)^{-1}x_{i}x_{i}'(X'X)^{-1}}{1-{x_{i}'(X'X)^{-1}x_i}}$$ Here is a proof that $h_{ii} = x_{i}'(X'X)^{-1}x_i$. If you multiply both sides by $x_i$ you get: $$(X_{(i)}'X_{(i)})^{-1}x_i=\frac{(X'X)^{-1}x_{i}}{1-h_{ii}}$$ which we are going to use both of these later. Now we find a formula for ${(\hat{\beta}_{(i)}-\hat{\beta})}$ : $${\hat{\beta}_{(i)}-\hat{\beta}}={(X_{(i)}'X_{(i)})^{-1}X_{(i)}'Y_{(i)}-\hat{\beta}}=(X_{(i)}'X_{(i)})^{-1}X'Y-(X_{(i)}'X_{(i)})^{-1}x_{i}y_{i}-\hat{\beta}={\frac{(X'X)^{-1}x_{i}x_{i}'\hat{\beta}}{1-h_{ii}}-(X_{(i)}'X_{(i)})^{-1}x_{i}y_{i}}={\frac{(X'X)^{-1}x_{i}}{1-h_{ii}}(x_{i}'\hat{\beta}-y_{i})}$$ Substitute this in $D_i$: $$D_i=\frac{r_i^2h_{ii}}{ps^2(1-h_{ii})^2}$$ where $r_i=(y_{i}-x_{i}'\hat{\beta})$.
