2
$\begingroup$

I read in more than one place that residual standard deviation can differ at different points on X. I cannot understand this statement. I find this while learning the very basics, so for me the standard deviation of the residuals is a single unique number, and I see no way how this single number can change depending on the xi being a simple number calculated across all samples. I guess I must be missing some necessary intermediate concept/steps. I would be very grateful is someone can explain this to me in a very basic way


My question was not clear trying to improve it: what is not clear to me is if there is such a thing as the standard-deviation of a specific point/sample, ie. if we have x <- 1:5, y <- c(10,6,12,15,2) and y ~ x, is there a formal concept of standard deviation of residual of a single sample and a formula to calculate it ? ie. can I, and if yes how do I calculate the standard deviation of the residual corresponding to x=1,y=10 ? of that corresponding to x=2,y = 6? ...

$\endgroup$
4
  • 1
    $\begingroup$ Your question concerns the distinction between errors and residuals. It can be informative to examine an extreme case. Consider fitting a line to the data $(0,\epsilon_1), (0,\epsilon_2), (100,\epsilon_3)$, where the $\epsilon_i$ are identically distributed errors and thereby have the same variances. The fit will pass right through that last point and split the first two points. Thus, the residuals $(\epsilon_2-\epsilon_1)/2, (\epsilon_1-\epsilon_2)/2,$ and $0$ will vary for the first two points but will never vary for the last point. $\endgroup$
    – whuber
    Feb 2 '17 at 21:47
  • $\begingroup$ You speak about residuals, no problem with them being different, that is what I would usually expect. My doubt is about how different observations, within the same sample, can have different standard deviations, this is what my course literally says. $\endgroup$
    – user110848
    Feb 10 '17 at 12:42
  • 1
    $\begingroup$ Data will be whatever they will be, so it's pointless to opine about what their values are. But the distributions of the residuals must differ even though the distributions of the errors are assumed not to. In the example I gave, the residual at $x=100$ is invariably zero while the residuals at $x=0$ clearly vary. Thus their distributions must differ with different $x$. $\endgroup$
    – whuber
    Feb 10 '17 at 13:48
  • $\begingroup$ My previous comment was wrong due to having moved my mind to something else and rereading in a hurry forgot that I had actually asked about residual variance, and not variance of observations. Sorry about that. Thanks for you comments/explanations. $\endgroup$
    – user110848
    Feb 10 '17 at 14:02
3
$\begingroup$

EDIT :

My answer was indeed wrong (thank you @Glen_b). I cannot delete it as it has been validated.

All I can do is to redirect you to this answer written by Alecos Papadopoulos.

In short, if you have a simple model: $y_i = \beta_0 + \beta_1*x_i + u_i$ then your $ith$ residual will be like $\hat{u_i} = y_i - \hat{y_i} = (\beta_0 - \hat{\beta_0}) + (\beta_1 - \hat{\beta_1})*x_i + u_i$.

If $Var(u_i) = \sigma$ then $Var(\hat{u_i}) = \sigma(1-\frac{1}{n} - \frac{(x_i - \bar{x})^2}{\sum(x_i^2-\bar{x}^2)})$.

All the calculation are explained in the link, and I could not do any better.

$\endgroup$
3
  • 1
    $\begingroup$ Thanks for your answer, I see now that my question was not clear, I try again If I understand well you use the term deviation like a synonim of residual. What is not clear to me is if there is such a thing as the standard-deviation of a specific point/sample, ie. if we have x <- 1:5, y <- c(10,6,12,15,2) and y ~ x, is there a formal concept of standard deviation of residual of a single sample and a formula to calculate it ? ie. can I, and if yes how do I calculate the standard deviation of the residual corresponding to x=1,y=10 ? of that corresponding to x=2,y = 6? ... $\endgroup$
    – user110848
    Feb 2 '17 at 17:42
  • $\begingroup$ I think my edition answers your question now. $\endgroup$
    – LouisBBBB
    Feb 3 '17 at 16:17
  • 2
    $\begingroup$ There's a very important issue that @whuber touches on that you seem not to be dealing with at all -- indeed some of what you say implies that you don't realize it happens -- but I think it's directly relevant here. If you use a linear regression on a random variable that is homoscedastic (errors have constant variance), the residuals will actually be heteroscedastic (the residuals will not have constant variance). A number of answers on site show calculations for the variance of residuals. $\endgroup$
    – Glen_b
    Feb 5 '17 at 23:52
0
$\begingroup$

I may have found a sort of answer. In some cases (analysis of residuals, this terminology might be unprecise) standard deviation is calculated excluding a sample/data-point. The standard deviation so calculated, though calculated over all samples (minus one, "that" sample/data-point), is uniquely "identified" by the sample/data-point due to its exclusion from the computation. (A similar thing seems to happen for standard error calculated for similar purposes)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy