11
$\begingroup$

I have a linear regression that's quite good, I guess (it's for a university project so I don't really have to be super accurate).

Point is, if I plot the residuals vs. predicted values, there is (according to my teacher) an hint of heteroskedasticity.

But if I plot the Q-Q-Plot of the residuals, it's clear that they are normally distributed. Moreover, the Shapiro test on the residuals has a $p$-value of $0.8$, so I think there's no doubt the residuals are actually normally distributed.

Question: How can there be heteroskedasticity on predicted values if the residuals are normally distributed?

$\endgroup$
  • 1
    $\begingroup$ One very short addition to @whubers excellent answer: You could use the ncvTest function of the car package for R to conduct a formal test for heteroscedasticity. In whuber's example, the command ncvTest(fit) yields a $p$-value that is almost zero and provides strong evidence against constant error variance (which was expected, of course). $\endgroup$ – COOLSerdash Jul 10 '13 at 20:08
15
+50
$\begingroup$

One way to approach this question is to look at it in reverse: how could we begin with normally distributed residuals and arrange them to be heteroscedastic? From this point of view the answer becomes obvious: associate the smaller residuals with the smaller predicted values.

To illustrate, here is an explicit construction.

Figure

The data at the left are clearly heteroscedastic relative to the linear fit (shown in red). This is driven home by the residuals vs predicted plot at the right. But--by construction--the unordered set of residuals is close to normally distributed, as their histogram in the middle shows. (The p-value in the Shapiro-Wilk test of normality is 0.60, obtained with the R command shapiro.test(residuals(fit)) issued after running the code below.)

Real data can look like this, too. The moral is that heteroscedasticity characterizes a relationship between residual size and predictions whereas normality tells us nothing about how the residuals relate to anything else.


Here is the R code for this construction.

set.seed(17)
n <- 256
x <- (1:n)/n                       # The set of x values
e <- rnorm(n, sd=1)                # A set of *normally distributed* values
i <- order(runif(n, max=dnorm(e))) # Put the larger ones towards the end on average
y <- 1 + 5 * x + e[rev(i)]         # Generate some y values plus "error" `e`.
fit <- lm(y ~ x)                   # Regress `y` against `x`.
par(mfrow=c(1,3))                  # Set up the plots ...
plot(x,y, main="Data", cex=0.8)
abline(coef(fit), col="Red")
hist(residuals(fit), main="Residuals")
plot(predict(fit), residuals(fit), cex=0.8, main="Residuals vs. Predicted")
$\endgroup$
  • 2
    $\begingroup$ ok so you're saying that if I associate low residuals with high predicted values, heteroscedasticity can arise even if residuals are normally distributed? I think I've got it, though i should really think more about it.. anyhow thank you! $\endgroup$ – Ant Jul 10 '13 at 21:17
  • $\begingroup$ ...or low residuals with low predicted values (as in the example here), or even in more complex ways. For instance, heteroscedasticity exists when the average magnitude of residuals oscillates with the predicted value. (Most formal tests of heteroscedasticity will not detect this, but the usual diagnostic plots will clearly show it.) $\endgroup$ – whuber Jul 10 '13 at 22:11
0
$\begingroup$

In weighted least squares (WLS) regression, it is the random factors of the estimated residuals that you might want to be able to see are normally distributed,  though it often isn't terribly important. The estimated residuals may be factored, as shown in a simple ( one regressor and through the origin) regression case, at the bottom of page 1, and bottom halves of pages 2 and 7 in https://www.researchgate.net/publication/263036348_Properties_of_Weighted_Least_Squares_Regression_for_Cutoff_Sampling_in_Establishment_Surveys Anyway, this might help show where normality can come into the picture.

$\endgroup$
  • $\begingroup$ Welcome to the site, @JimKnaub. We'd love to have you around to lend your expertise on the occasional question. Why not register your account? You can find out how in the My Account section of our help center. Since you're new here, you may want to take our tour, which has information for new users. $\endgroup$ – gung Jan 26 '16 at 0:09
  • 3
    $\begingroup$ We are trying to build a permanent repository of high-quality statistical information in the form of questions & answers. Thus, we're wary of answers that depend on links, due to linkrot. Can you post a full citation & a summary of the information (eg, figures / explanations) from the link so the information will stay useful even if the link goes dead? $\endgroup$ – gung Jan 26 '16 at 0:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.