# Linear Algebra Treatment of Cook's Distance (or any situation involving deleting an observation)

I'm having trouble reconciling the idea of "deleting" a fitted value from a vector of fitted values in order to calculate the Cook's Distance. This notion is also troubling me in all of the similar sorts of diagnostics.

in order to calculate the Cook's distance I do the following:

$$D_{i} = \frac{(\hat{Y} - \hat{Y_{(i)}})^{t}(\hat{Y} - \hat{Y_{(i)}})}{p \cdot MSE}$$

Where $$\hat{Y_{(i)}}$$ is the deleted case from a vector of $$n$$ observations. So if I "delete" a predicted value that means $$\hat{Y}$$ and $$\hat{Y_{(i)}}$$ are of different dimensions. I'm getting this from the concept that a vector of $$n$$ predicted values is represented by:

$$\hat{Y} = \begin{bmatrix} \hat{Y_{1}} \\ \vdots \\ \hat{Y_{i}} \\ \vdots \\ \hat{Y_{n}} \end{bmatrix}$$

removing the $$i^{th}$$ case leaves me with one vector of $$n$$ values and one of $$n-1$$ values.... so how would the algebra actually work?

The whole motivation for these questions comes from a question where I'm asked to prove the equality of the above Cook's Distance above with an alternative form:

$$D_{i} = \frac{(b - b_{(i)})^{t}X^{t}X(b - b_{(i)})}{p \cdot MSE}$$

Where $$b$$ is the vector of regression coefficients. I can do the "mechanical" algebra to solve and show the equality, but that is just through the guise of symbol manipulation. In attempting to understand/visualize what is really going on is where I am having issue. In terms of how I'm viewing the dimensions of the sets of vectors I have the following:

$$\underset{n \times 1}{\hat{Y}} = \underset{n \times p}{X}\ \underset{p \times 1}{b} \\ \underset{n - 1 \times 1}{\hat{Y_{(i)}}} = \underset{n - 1 \times p}{X_{(i)}}\ \underset{p \times 1}{b_{(i)}}$$

Now taking their difference to use in an expression such as Cook's Distance:

$$\underset{n \times 1}{\hat{Y}} - \underset{n - 1 \times 1}{\hat{Y_{(i)}}}$$

But as you can see they are of length $$n$$ and $$n-1$$ respectively.

I think part of my issue is in how I'm viewing the $$X$$ matrix. I'm thinking that we delete a row of the $$X$$ matrix to obtain the result for the "deleted" case. If I don't "delete" this case and just leave the $$X$$ matrix as is after some manipulation I'll get the result.

I know this seems trivial in the grand scheme of things and I was hoping not to have to write a full blown question about it, but I'm attempting to do some exercises and the idea is coming up when I think of things.

EDIT: Added more context, hopefully this clarifies my misunderstanding.

• I think your understanding is off just a bit. Cook's Distance for observation $i$ is the normalized sum of changes in predictions for all other observations if observation $i$ were removed from the data. So, $\hat{Y}_j - \hat{Y}_{j(i)}$ is the difference between the predictions for observation $j$ if $i$ was in the model and if $i$ was not in the model. Jul 10, 2021 at 23:14
• Actually I'm rewriting the question to provide some more detail of where I'm having issue. Give me a few mins. Jul 10, 2021 at 23:16
• @JasonMorgan I rewrote things to provide more context....hopefully this makes sense of my issue now. I do understand what Cook's distance is doing and how it is calculated theoretically, but actually doing it is bringing the issues Jul 10, 2021 at 23:44

$$\hat Y_{(i)}$$ doesn't mean what you think it means. It's an $$n\times 1$$ vector whose $$i$$th element is $$X_ib_{(i)}$$.
That is, the $$i$$th element of $$\hat Y-\hat Y_{(i)}$$ is $$X_i(b-b_{(i)})$$, the difference between the prediction for $$Y_i$$ when $$Y_i$$ is in the fitting data and the prediction for $$Y_i$$ when $$Y_i$$ is not in the fitting data.
• Ah...I see what you're saying. The REGRESSION MODEL for $\hat{Y_{(i)}}$ is obtained from using the $n-1$ remaining rows of my data set. Then using this model I predict the value of $Y_{i}$ by using the data points from the deleted observation, that I had previously removed to build my model. Jul 13, 2021 at 4:12