I implemented a linear regression model on some dataset. When I plotted the scatter plot of residual v/s predicted y (i.e., yhat), I observed heteroscedasticity in the plot. What can I do about it?
x-axis -> yhat
y-axis -> Residual
$\begingroup$
$\endgroup$
9
-
$\begingroup$ Would you please post a link to the data? $\endgroup$– James PhillipsSep 29, 2019 at 21:41
-
$\begingroup$ I am sorry. I can't $\endgroup$– user5001880Sep 29, 2019 at 21:42
-
1$\begingroup$ It clearly looks that as the yhat goes away from zero, the variance reduces. I can't think of a proper function between variance and yhat which I can use for variance stabilization method $\endgroup$– user5001880Sep 29, 2019 at 21:44
-
1$\begingroup$ Does a similar plot of residual error vs. actual y have a similar appearance? $\endgroup$– James PhillipsSep 29, 2019 at 23:11
-
2$\begingroup$ Without some context, not much useful can be said. What are your variables measuring? how many predictors? But the symmetric look is intriguing, and the fact that variance is largest around predicted values of zero is also interesting! But: The obvious overplotting in the plot hides information, maybe, so: How many observations in total? Maybe make a plot with smaller plotting symbols, lower alpha to get transparency, ... $\endgroup$– kjetil b halvorsen ♦Sep 30, 2019 at 12:27
|
Show 4 more comments
2 Answers
$\begingroup$
$\endgroup$
6
Before doing anything about heteroscedasticity, first you must establish whether it even exists.
The scatterplot in the question is not evidence of heteroscedasticity.
"Heteroscedasticity" refers to any important variation in the dispersion of one variable (here plotted on the vertical axis) when another variable (here plotted on the horizontal axis) is varied.
The problem with this plot is that it does not enable us to make accurate estimates of the dispersion. The reason why not is that because there are so many points, essentially all we can see are the ranges of the vertical variable, but the range is both a highly unstable estimate of dispersion as well as dependent on the amount of data. As you scan left to right across this plot, it is possible for there to be much, much more data in the middle compared to the ends. That can give rise to a false impression of greater dispersion of $y$ for middling $x$-values.
To illustrate this problem, I generated 400,000 $(x,y)$ values where the coordinates are independent--and therefore $y$ is as homoscedastic (non-heteroscedastic) as possible.
n <- 4e5
x <- pmax(-8, pmin(8, abs(rnorm(n, sd=7/6))^1.25 * sample(c(-1,1), n, replace=TRUE)))
y <- sample(c(-1,1), n, replace=TRUE) * rgamma(n, 1/3, 1/20)
Whether or not you are familiar with R (the language used for this simulation), it should be apparent that the x values and y values have been separately generated. But here's their scatterplot, shown using the overplotting method of the question:
It sure looks like the y-values are spread out less near the extremes than in the middle, doesn't it? This mistaken impression is the cause of many related questions (and incorrect answers) on this site.
A better way to look at heteroscedasticity is to split the $x$ values into groups and robustly estimate the dispersion of each group. There are many ways to do this. John Tukey observed that generally there's not a whole lot going on in the middle of a batch of data: the interesting changes tend to occur out in the extremes. In his "wandering schematic plot" he proposed splitting the $x$ data (essentially) into the most extreme halves, quarters, eighths, and so on, and plotting some of the extremes of the $y$ values found within each such group. This leads to a quick, simple plot.
The vertical bars in this figure demark the boundaries between the groups of $x$ values used to estimate the dispersion of the corresponding $y$ values. Thus, the middle half of all points falls within the two vertical strips surrounding $x=0.$ (Each of those strips contains about 100,000 points.) Thereafter, as you move away from $0,$ each vertical strip includes one-half as many points as previously. Ultimately the outer strips (roughly from $x=-8$ to $-5.4$ and $x=5.4$ to $8$) contain only $200$ points each.
The horizontal black graphs trace out the 1%, 5%, 25%, 50%, 75%, 95%, and 99% quantiles of the corresponding $y$ values. We can't distinguish the middle quantiles because they are so close, but it's nevertheless clear that these "wandering traces" are essentially horizontal. This shows, contrary to what the original figure suggested, that the dispersion of $y$ does not appreciably vary with $x.$ Indeed, it shows that the entire (conditional) distribution of $y$ appears to be independent of $x.$ That's what you're looking for when you assess the relationship between regression residuals and predicted values.
What does heteroscedasticity look like? Here's an example. The preceding $y$ values were divided by the larger of $1$ and $x^2,$ thereby reducing their dispersion (greatly) near the extremes of $x.$ The schematic traces are no longer horizontal: they contract towards the left and right to reflect the decreasing dispersion with large values of $x.$
(This time I plotted only a random sample of 10,000 points, because plotting all 400,000 requires a wait. But the traces are based on all 400,000 points.)
Finally, I suspect you might be correct that there is heteroscedasticity: but before we can say anything about what to do, it still needs to be represented clearly, quantified, and accurately characterized. Creating a wandering schematic plot or one of its newer variants (such as quantile regression) would be a good first step.
References
For an example of the wandering schematic plot used to diagnose heteroscedasticity, see https://stats.stackexchange.com/a/166267/919. Other examples of this plot can be found at https://stats.stackexchange.com/a/106083/919 (with working R code) and Measures of residuals heteroscedasticity (also with code).
Tukey describes this method in his book Exploratory Data Analysis (Addison-Wesley 1977). He provides many more ways to depict and diagnose heteroscedasticity, as well as methods to "cure" it, such as Box-Cox transformations.
-
$\begingroup$ Thanks. Actually I thought of the somewhat same idea before I looked at your answer. I was thinking about slicing the x's into vertical boxes and then in each box find the variance of y. But I think instead of variance if we plot the box plots for each slice i.e. wandering schematic plot, it will provide more insights $\endgroup$ Sep 30, 2019 at 20:00
-
$\begingroup$ On a side note, I think box-cox transformation is only valid when the target variable is positive because in box-cox we apply transformation f(y) = (y^lambda - 1)/lambda where lambda is any real number. So what shall be done for the case when the target variable can be both negative and positive $\endgroup$ Sep 30, 2019 at 20:00
-
$\begingroup$ The target variable is the original response in the regression, not the residual. Also, as illustrated at stats.stackexchange.com/a/35717/919 (inter alia), there are principled ways to shift variables to make them positive and apply a Box-Cox transformation. There are also many nonlinear transformations available for variables generally (see stats.stackexchange.com/questions/251449 for a general analysis); my reference to the Box-Cox family was only an example, not a recommendation. $\endgroup$– whuber ♦Sep 30, 2019 at 20:11
-
1$\begingroup$ The idea of that plot actually helped. Here is the plot of 20 box-plots. I divided the yhat(i.e. x-axis in the plot in the question) into 20 slices and made a box plot of data for each of the slices $\endgroup$ Sep 30, 2019 at 21:14
-
$\begingroup$ drive.google.com/file/d/1_DOZh4SJiU1snpaAHybD-BB7qb0kW0_h/… This looks like a straight line i.e. residual is directly proportional to yhat. which means that, the regression hasn't happened at all. And that is my the R^2 value is comint as 0.029 $\endgroup$ Sep 30, 2019 at 21:29
$\begingroup$
$\endgroup$
1
The existence of heteroscedasticity gives rise to certain problems in the regression analysis as the assumption says that error terms are uncorrelated and, hence, the variance is constant. The presence of heteroscedasticity can often be seen in the form of a cone-like scatter plot for residual vs fitted values.
One of the basic assumptions of linear regression is that heteroscedasticity is not present in the data. Due to violation of the assumptions, the Ordinary Least Squares (OLS) estimators are not the Best Linear Unbiased Estimators (BLUE). Hence, they do not give the least variance than other Linear Unbiased Estimators (LUEs).
There is no fixed procedure to overcome heteroscedasticity. However, there are some ways that may lead to the reduction of heteroscedasticity. They are —
Logarithmising the data: A series that is increasing exponentially often results in increased variability. This can be overcome using the log transformation.
Using weighted linear regression: Here, the OLS method is applied to the weighted values of X and Y. One way is to attach weights directly related to the magnitude of the dependent variable.
-
2$\begingroup$ Your first paragraph reads a bit like saying that heteroscedasticity implies the errors are correlated. $\endgroup$ Sep 30, 2019 at 15:54