Is there a specific standard error for an individual residual?

Question

On page 97 of Introduction to Statistical Learning book, there is a paragraph on studentized residuals within the context of looking for outliers.

But in practice, it can be difficult to decide how large a residual needs to be before we consider the point to be an outlier. To address this problem, instead of plotting the residuals, we can plot the studentized residuals, computed by dividing each residual e_i by its estimated standard error.

This gist of this makes sense, I understand that we are looking for residuals that are more than 2-3 deviations away from the mean.

What confused me was use of the word "its"

in each residual e_i by its estimated standard error

This implies there is a standard error for each residual e_i as opposed to a standard error for all of the residuals as a group. Have I misread or misunderstood this?

When calculating studentized residuals am I dividing each residual by the regression RSE or some other denominator?

Answer 1

Yes, at least for linear regression, so I will answer for that case. The model is then $$ Y=X\beta+\epsilon $$ in matrix form, where $Y$ is $n\times 1$, $X$ is $n\times p$, $\beta$ is $p\times 1$ and the error term $\epsilon$ is $n\times 1$. We assume the errors $\epsilon_i$ are iid with variance $\sigma^2$ (and expectation zero).

Then the least squares estimator is $\hat{\beta}=(X^T X)^{-1}X^T Y$ and the predicted values is $\hat{Y}= X\hat{\beta}=X (X^TX)^{-1} X Y= HY$ where $H= X (X^TX)^{-1} X$ is the hat matrix. The residual vector $r= Y-\hat{Y}=(I-H)Y$ has expectation zero and variance matrix $\text{Var}(r)= (I-H) \sigma^2 I (I-H) = \sigma^2 (I-H)$ so the standard error of residual component $i$ is $$ \text{se}(r_i) = \sigma \sqrt{1-h_i} $$ and we can show that $ 0\le h_i \le 1$. $h_i$ is often seen as a measure of the influence or leverage of observation $i$, so we can see that the most influential observations have residuals with lesser standard error.