5
$\begingroup$

In order to assess the "correctness" of a specified regression model, one can compute the variance of the residuals and compare it to the uncertatinty $\sigma$ of our data. To do so, it is useful to work with standardized/studentized Residuals.

Assuming Homoskedasticity we get:

Standardized Residuals: $$ R_{i,Stand} = \frac{R_i}{\hat{\sigma}}$$ Internally studentized Residuals: $$ R_{i,ExtStud} = \frac {R_i}{var(R_i)} = \frac{R_i}{\sqrt{\hat{\sigma}^2 (1 - H_{ii})}}$$

Externally studentized Residuals (follow a t-distribution): $$ R_{i,IntStud} = \frac{R_i}{\sqrt{\hat{\sigma}_{(i)}^2 (1 - H_{ii})}} \sim t(n-p-1) \text{-distribution}$$

Now I found some literature stating that the $Var(R_{Stand})$ should equal 1, and some literature stating that $Var(R_{IntStud})$ or $Var(R_{ExtStud})$ should equal 1.

Which one is it ?

$\endgroup$
2
  • $\begingroup$ plug both to the variance equation and see, think of what is $\hat\sigma$ $\endgroup$
    – Aksakal
    Commented May 15, 2020 at 12:51
  • $\begingroup$ You mean $Var(X) = E[X^2] - E[X]^2$ ? I know that $\hat{\sigma} = \sqrt{\frac{\sum R_i^2}{\nu}}$ and $\frac{(n-p-1)\hat{\sigma} }{\sigma^2}\sim \chi^2_{n-p-1}.$ But I am stuck at this point... $\endgroup$ Commented May 15, 2020 at 20:56

3 Answers 3

4
$\begingroup$

If you're talking about the variance of the random variable (taken with the explanatory variables fixed and the response as a random variable) then none of these have exactly unit variance. What we can legitimately say under the linear regression model with OLS etimation is that:

$$\begin{align} \mathbb{V}(R_i | \mathbf{x}) &= \sigma^2 (1-h_{i,i}), \\[6pt] \mathbb{E}(\hat{\sigma}^2 | \mathbf{x}) &= \sigma^2, \\[6pt] \mathbb{E}(\hat{\sigma}_{(i)}^2 | \mathbf{x}) &= \sigma^2. \\[6pt] \end{align}$$

Moreover, under broad limiting conditions we also know that there is convergence $\hat{\sigma}_{(i)}^2 \rightarrow \hat{\sigma}^2 \rightarrow \sigma^2$ as $n \rightarrow \infty$. From these results we can show that the variance of the standardised/studentised residuals approaches one asymptotically under the required limiting conditions. However, from these results, it does not follow that any of the ratios you give have exact unit variance for any finite $n$.

In fact, the only case where we can calculate an exact distribution (and an exact variance) that is independent of the explanatory variables is for the externally studentised residuals,$^\dagger$ where we have:

$$R_{i,\text{ExtStud}} = \frac{R_i}{\sigma_{(i)} \sqrt{1-h_{i,i}}} \sim \text{St}(n-p-1).$$

In this case, using the variance of the Student's T distribution we have the exact variance:

$$\mathbb{V}(R_{i,\text{ExtStud}} | \mathbf{x}) = \frac{n-p-1}{n-p-3} \quad \quad \quad \quad \quad \text{if } n-p>3.$$

We can see that $\mathbb{V}(R_{i,\text{ExtStud}} | \mathbf{x}) \rightarrow 1$ as $n \rightarrow \infty$, but the variance is not one for any finite $n$. In the other two cases the exact conditional distribution of the residual (given the explanatory variables) is complicated and depends on the explanatory variables. However, as previously noted, we have convergence to unit variance in all cases under broad limit conditions.


$^\dagger$ You have your notation for internal and external studentisation around the wrong way; it is external if the variance is estimated without use of the present data point.

$\endgroup$
3
$\begingroup$

The internally studentized residuals do have exactly unit variance.

Consider a linear regression model $\boldsymbol y=X\boldsymbol\beta+\boldsymbol\varepsilon$, where $\boldsymbol y$ is an $n\times 1$ response vector, $X$ is an $n\times p$ matrix of covariates (fixed), $\boldsymbol \beta$ is a $p\times 1$ vector of parameters, and the error vector $\boldsymbol\varepsilon$ is multivariate normal $N_n(\boldsymbol 0,\sigma^2I)$.

The $i$th internally studentized residual is

$$r_i=\frac{e_i}{\hat\sigma\sqrt{1-h_{ii}}}\,,$$

where $e_i=y_i-\boldsymbol x_i^T\hat{\boldsymbol\beta}$ is the $i$th residual, $h_{ij}$ is the $(i,j)$th entry of the hat matrix $H=X(X^TX)^{-1}X^T$, and $\hat\sigma^2=\frac1{n-p}\sum_{j=1}^n e_j^2$ is the usual unbiased estimator of $\sigma^2$. Also, $\boldsymbol x_i^T$ is the $i$th row of $X$ and $\hat{\boldsymbol\beta}$ is the least squares estimate of $\boldsymbol\beta$.

Note that $e_i \sim N(0,\sigma^2(1-h_{ii}))$ for each $i$.

The mean of each $r_i$ is $0$ because $e_i/\hat\sigma$ is symmetric about $0$. So the variance is just the second moment, which one can find using the distribution of $r_i^2$:

$$\frac{r_i^2}{n-p}\sim \text{Beta}\left(\frac12,\frac{n-p-1}{2}\right) \tag{1}$$

So, $$\operatorname{Var}(r_i)=(n-p)\operatorname E\left[\frac{r_i^2}{n-p}\right]=\frac{(n-p)/2}{1/2+(n-p-1)/2}=1$$


For a simple derivation of $(1)$, we can use the relationship between $\hat\sigma^2$ and $s_{(i)}^2=\frac1{n-p-1}\sum\limits_{j(\ne i)=1}^n \left(y_j-\boldsymbol x_j^T\hat{\boldsymbol\beta}_{(i)}\right)^2$, where $\hat{\boldsymbol\beta}_{(i)}$ is the least squares estimate of $\boldsymbol\beta$ with the $i$th case removed.

First we need the following formula for $\operatorname{DFBETA}_i$:

$$\operatorname{DFBETA}_i := \hat{\boldsymbol\beta}-\hat{\boldsymbol\beta}_{(i)} = \frac{(X^TX)^{-1}\boldsymbol x_i e_i}{1-h_{ii}} \tag{2}$$

Then,

\begin{align} (n-p-1)s_{(i)}^2 &= \sum_{j(\ne i)=1}^n \left[(y_j-\boldsymbol x_j^T\hat{\boldsymbol\beta})+\boldsymbol x_j^T(\hat{\boldsymbol\beta}-\hat{\boldsymbol\beta}_{(i)})\right]^2 \\&=\sum_{j(\ne i)=1}^n \left[e_j+\frac{h_{ji}e_i}{1-h_{ii}}\right]^2 \\&=\sum_{j=1}^n \left[e_j+\frac{h_{ij}e_i}{1-h_{ii}}\right]^2 - \left[e_i+\frac{h_{ii}e_i}{1-h_{ii}}\right]^2 \\&=\sum_{j=1}^n e_j^2 + \frac{e_i^2}{(1-h_{ii})^2}h_{ii} - \frac{e_i^2}{(1-h_{ii})^2} \\&=(n-p)\hat\sigma^2 - \frac{e_i^2}{1-h_{ii}} \end{align}

In the penultimate step, we have used $h_{ii}=\sum_{j=1}^n h_{ij}^2$ and $\sum_{j=1}^n h_{ij}e_j=0$, which follow from $H=H^2$ and $H\boldsymbol e=\boldsymbol 0$ respectively.

Now $(\star)$ follows from

\begin{align} \frac{r_i^2}{n-p}&=\frac{e_i^2/(1-h_{ii})}{(n-p)\hat\sigma^2} \\&=\frac{\frac{e_i^2}{\sigma^2(1-h_{ii})}}{\frac{(n-p-1)s_{(i)}^2}{\sigma^2}+\frac{e_i^2}{\sigma^2(1-h_{ii})}} \\&=\frac{U}{U+V}\,, \end{align}

where $U=\frac{e_i^2}{\sigma^2(1-h_{ii})}\sim \chi^2_1$ and $V=\frac{(n-p-1)s_{(i)}^2}{\sigma^2}\sim \chi^2_{n-p-1}$ are independently distributed.


For proving $(2)$, we first define $X_{(i)}$ and $\boldsymbol y_{(i)}$ as the $X$ matrix and $\boldsymbol y$ vector without their $i$th rows.

Then $$\hat{\boldsymbol\beta}_{(i)} =(X_{(i)}^TX_{(i)})^{-1}X_{(i)}^T\boldsymbol y_{(i)}$$

Now using

$$X^TX=X_{(i)}^TX_{(i)}+\boldsymbol x_i\boldsymbol x_i^T$$

and $$X^T\boldsymbol y=X_{(i)}^T\boldsymbol y_{(i)}+\boldsymbol x_i y_i$$

combined with the Sherman-Morrison formula leads to

$$\hat{\boldsymbol\beta}_{(i)}=\hat{\boldsymbol\beta}-\frac{(X^TX)^{-1}\boldsymbol x_i e_i}{1-h_{ii}}$$

$\endgroup$
1
  • 2
    $\begingroup$ This looks good. (+1) One can also refer to Weisberg's Residuals and Influence in Regression, specifically the first chapter as they cover the same in a rigorous way. $\endgroup$ Commented May 12, 2023 at 8:39
1
$\begingroup$

When you standardize, it's a mechanical operation, there's no $\chi^2$ and other fun stuff. It's just $r_i=\frac{R_i-\bar R_i}{s^2}$, hence, if you take a variance of this thing, then you get $var[r_i|R]=\frac{var{R_i|R}}{s^2}=\frac{s^2}{s^2}=1$

Here, $s^2$ is a sample variance, and although it's in denominator, it does NOT lead to $\chi^2$ because it's in a conditional expression. It's great because for $\chi^2$ you need normal assumptions, and here we don't need any distributional assumption whatsoever. We don't even need the formulae I have given here, because you can derive the same result from simply arithmetic, no probability theory is needed at all

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.