7
$\begingroup$

In my course notes on a regression course with regards to the detection of heteroskedasticity there's the following quote:

"Because the least-squares residuals have unequal variances even in the homoscedastic case, it is preferable to use the standardized residuals."

My intuition tells me that since the LS regression line necessarily goes through the center of the datacloud, it will be a better fit for points in the middle of the covariate space than on the tails, thus giving us larger variance on the extremes.

Despite this, this does not seem like it's necessary. And at the same time I wonder about why do we care for homoscedasticity on standardized or studentized residuals and not for the raw ones.

$\endgroup$
  • 3
    $\begingroup$ For residuals, the variance is smaller at the extremes. This is because the more extreme observations have more influence over the regression function. ('more extreme' = further from the mean in x-space, as measured by Mahalanobis' distance) [Specific formulas are readily derived or may be found in other answers on site.] $\endgroup$ – Glen_b -Reinstate Monica Apr 18 at 3:18
  • $\begingroup$ The answer can be found here: stats.stackexchange.com/questions/212656/… $\endgroup$ – kjetil b halvorsen Apr 18 at 21:46
  • 1
    $\begingroup$ There's a derivation of the variance of a residual for the multiple regression case and some additional explanation here $\endgroup$ – Glen_b -Reinstate Monica Apr 19 at 1:22
4
$\begingroup$

Assuming the usual linear model with constant variance $\sigma^2$. I will use notation (and some results) from Leverages and effect of leverage points. The linear model in matrix form is $$ Y= X\beta + \epsilon $$ where $\epsilon$ is a vector of $n$ iid error terms. Then the hat matrix is $H=X(X^TX)^{-1}X^T$, and its diagonal terms are the leverages $h_{ii}$. We can show that the variance of the residuals $e_i = y_i-\hat{y_i}$ is $\sigma^2 (1-h_{ii})$ (remember $0<h_{ii}<1$.)

So, under this model, to get constant-variance residuals we divide by $\sqrt{1-h_{ii}}$: the standardized residuals defined by $r_i=\frac{y_i-\hat{y}_i}{\sqrt{1-h_{ii}}}$ have constant variance. So for many uses in residuals analysis we prefer this standardized residuals, for instance in checking the assumption of constant variance.

EDIT

In a comment the OP writes:

As far as I know the formal assumption is not "homoscedasticity of standardized residuals", but only residuals by itself.

This confuses errors with residuals. The errors are the unobserved $\epsilon_i$ in the regression equation $y_i =\beta_0 +\sum_i \beta_i x_i +\epsilon_i$, while residuals is the observed difference between observation and model prediction. Homoskedastcity means that the the errors all have the same variance, not that the residuals have constant variance. If you want to use residuals to test/critizise the constant variance assumption, it is better to use a version of the residuals that do have constant variance (under the model.)

$\endgroup$
  • $\begingroup$ With regards to the second part of my question, I can understand why it is more convenient to use standardized residuals, but my question I think is more abstract: why the natural heteroskedasticity does not disturb by itself our Gauss-Markov conditions and standard error estimates. As far as I know the formal assumption is not "homoscedasticity of standardized residuals", but only residuals by itself. Aren't we just "covering the sun with a finger"? $\endgroup$ – Kuku Jun 17 at 0:24
1
$\begingroup$

Suppose you have three $x$-values: $-1,0, +1.$

The corresponding dependent variables $Y_1,Y_2,Y_3$ are where the randomness is.

Now draw the picture. You can see why, if you move $Y_2$ up or down, the fitted line moves up or down. (By just $1/3$ as much as $Y_2$ moves.) But what happens if you move $Y_3$ up or down? The fitted line doesn't just move up or down; its slope also gets bigger or smaller. Or if you move $Y_1$ up or down, then the slope gets smaller or bigger, respectively. So the line has more tendency to stay close to the data point when the data point's $x$-value is far from the average $x$-value than when it is near the average $x$-value. Hence the observed residuals have a smaller variance when the $x$-value is far from the average $x$-value than when the $x$-value is close to the average $x$-value.

The fitted values are \begin{align} & \left(\widehat Y_1, \widehat Y_2, \widehat Y_3\right) \\[5pt] = {} & \left( \tfrac 2 3 Y_1+ \tfrac 1 3 Y_2, \,\,\, \tfrac 1 3 (Y_1+Y_2 + Y_3), \,\,\, \tfrac 1 3 Y_2 + \tfrac 2 3 Y_3 \right). \end{align} So the residuals are \begin{align} & \left( Y_1, Y_2, Y_3 \right) - \left(\widehat Y_1, \widehat Y_2, \widehat Y_3\right) \\[5pt] = {} & \left( \tfrac 1 3 Y_1 - \tfrac 1 3 Y_2, \,\,\, -\tfrac 2 3 Y_1+ \tfrac 2 3 Y_2 - \tfrac 2 3 Y_3, \,\,\, -\tfrac 1 3 Y_2 + \tfrac 1 3 Y_3 \right). \end{align} From this one can compute the variances of the residuals.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.