4
$\begingroup$

This is a residual vs. predictor plot for my regression problem. All the other residual plots don't show clear non-constant variance, but this one definitely stands out, and its variance is not monotonic as x42 increases.

I've tried variance-stabilizing transformations (square root and log) on y and it doesn't work, quite expectedly. What are other things I can try in this case?


Edit 02/15/2017: The problem at hand is a variable selection and OLS problem for a chemical reactor data set. I have more than 50 predictors that could possibly explain my single response $y$. $x_{42}$ is a predictor that gets picked by best subsets, and it represents the effects of catalysts added to the reactor. Initially (in time) there was no catalyst added, then there's some, and eventually full dose of catalyst is added every day. As a result, the "distribution" of $x_{42}$ takes that shape.

$x_{42}$ is not a random variable that should take any known or unknown statistical distribution, because it's a variable driven by engineering decisions. My education tells me the "distribution" of predictor doesn't matter in regression, so I didn't have doubts whether OLS applies to my data set with a variable like $x_{42}$.


Edit 02/16/2017: Let me further clarify my objective here. What I wanted to know is:

  1. Does this residual plot show heteroscedasticity?
    I'm inclined to agree with @mdewey that most of the points are at where $x_{42}=0$ or $x_{42}=750$ so it is expected that the apparent scatter of residuals to be larger there, and this doesn't necessarily imply the variance of the residuals is highly non-constant across the range of $x_{42}$. If there are other reliable tests that can help me better determine if there's heteroscedasticity, please kindly advise.

  2. If the amount of heteroscedasticity in this residual plot is so large that it could throw off my inference (p-value, CI, etc.), what are the remedies?
    As suggested by @whuber, no monotonic transformation on $y$ would cure it in this case, and I fully agree. What other options do I have? Bootstrap? GLM? I can try all the options, but it would be difficult to gauge which method is better. So if you could shed some light on which option is intrinsically more suitable, that is highly appreciated.

enter image description here

$\endgroup$
  • 3
    $\begingroup$ This has now attracted four answers which (a) do not seem to have much contact with one another, and (b) have attracted many down-votes. Perhaps you could edit your question with some more information about the problem you are facing and see whether that clarifies the issues? $\endgroup$ – mdewey Feb 13 '17 at 14:23
  • $\begingroup$ @mdewey I've edited my OP with more information of the problem. Please see if that helps. $\endgroup$ – hooyeh Feb 15 '17 at 19:34
  • $\begingroup$ Your expanded description notes "Initially (in time) there was no catalyst added, then there's some, and eventually full dose of catalyst is added every day". So really the three regimes should probably not be mixed up into a single regression. In terms of $x_{42}$, you can only reasonably expect a relationship over $(0,750)$. (Though you could estimate conditional distributions for $y|x_{42}=0$ and $y|x_{42}=750$, e.g. means and stdev's.) You would need some other predictor to untangle the rest (e.g. perhaps "time since start" and "time since $x_{42}$ @ full dose"). $\endgroup$ – GeoMatt22 Feb 16 '17 at 1:05
  • $\begingroup$ @GeoMatt22 Thanks for the comment. Two reasons that I didn't partition the data: 1. I believe the sensitivity of $y$ on $x_{42}$ is more-or-less constant, i.e., I don't expect $y$ to nonlinearly depend on $x_{42}$, and I can get a linear relationship between $y$ and other $x$s including $x_{42}$ that is applicable to $x_{42}$ being any value within $[0, 750]$. 2. Now my $x_{42}$ has three regimes, but a worse case would be $x_{42}$ being a strictly binary variable. But we frequently have binary variables in regression, since that's ok, I think it should also be ok for me here. $\endgroup$ – hooyeh Feb 16 '17 at 19:21
  • 3
    $\begingroup$ A simple exploratory test of heteroscedasticity is to plot the absolute values of the residuals against the predictions and smooth that plot (pretty heavily). Since that plot's not available, if we take x42 as a surrogate for the predictions, it seems there is little evidence in it of heteroscedasticity. Consider making appropriate plots of your data and studying them. $\endgroup$ – whuber Feb 16 '17 at 20:11
4
$\begingroup$

When you have multiple predictors in your model (as it sounds like you do), then you need to plot the residuals against the predicted values for Y, not against any given predictor. The assumption about homogeneity/homoscedasticity refers to the distribution of the observed values relative to the predicted values (i.e. the residuals). Here's a visual you might find helpful:

homoscedasticity

The assumption of homoscedasticity is that the variance of the distribution of the observations relative to their predictions (i.e. the regression line) is equal. In other words, the density plots depicted all have the same variance. In the example depicted there, there is only one predictor (making it easy to show the regression on one plane with just two axes). If there were multiple predictors, then the regression line would cut through k-dimensional space for k-1 predictors --- for 2 predictors, imagine a 3D cloud of points with the line of best fit cutting through it. If you look at the residuals relative to any one of those predictors, you're potentially looking at them from a weird angle. This can be especially confusing if one of your predictors is really weirdly distributed itself, as your x42 appears to be.

In order to see whether or not you have an issue with the homoscedastictiy of your residuals, you need to plot the residuals on the y-axis and the predicted values on the x-axis. In effect, this zooms in on the regression line itself --- no matter where it is in our hypothetical k-dimensional space --- and shows you the residuals relative to the regression line. I'm not sure what software you're using, but many will easy (or even automatically) produce such a plot for you.

If you do that and you still see a problem with the variance of your residuals, then you may want to consider WLS regression instead of OLS regression. It will give observations in lower-variance areas more weight in determining the regression coefficients, allowing for the fact that you apparently have better precision there. It also has the handy side effect of reducing the influence of potential outliers in the higher-variance parts of your data.

$\endgroup$
  • 1
    $\begingroup$ There is more to this than residuals. It is true that OLS will yield a least error estimate in Y. That can also be irrelevant when X is a random variable, and when the question is to find a relationship between X and Y, instead of a least error predictor of Y. See Deming regression. $\endgroup$ – Carl Feb 12 '17 at 19:40
  • 1
    $\begingroup$ Thanks, @Carl. There's nothing about OP's question that suggests the predictors are random variables, however, and the residuals OP chose to plot are just in Y, suggesting OLS is the technique of choice here. $\endgroup$ – Rose Hartman Feb 12 '17 at 20:19
  • $\begingroup$ The X-axis in the plot is approximately scaled beta distributed. Not only is there a random X, but also enough info to suggest what type. $\endgroup$ – Carl Feb 12 '17 at 20:36
  • 1
    $\begingroup$ Looking at individual predictors can give you more context on a potential problem you diagnose by looking at the residuals against the predicted values, but what I'm trying to emphasize is that if there's no problem in the residuals plotted against the predicted values, then you don't have an issue with heteroscedasticity. Looking at individual predictors can be a follow up, but it's a potentially misleading first step. $\endgroup$ – Rose Hartman Feb 17 '17 at 19:51
  • 2
    $\begingroup$ +1, this answer is correct, & constitutes the bulk of what the OP can (should) be told given the information in the body of the question, IMO. I do not understand the rationale for any downvote here. $\endgroup$ – gung Feb 19 '17 at 16:20
3
$\begingroup$

Looking at this sort of plot is very difficult because of the distribution of x42. Since most of its values are either zero or 750 the spread at those points is bound to look greater. If you want to see how the variance varies as a function of x42 I suggest plotting the square root of the absolute value of the residual and fitting a loess line. If that is essentially flat then all is well.

The more interesting question though is why x42 has such an unusual distribution and what hypothesis you can have about its relationship with any $y$ variable.

$\endgroup$
  • 1
    $\begingroup$ Could you explain why plotting the sqrt of the absolute value of the residual and fitting a loess line can be used to determine if the heteroscedasticity is of concern? If there's derivation, that helps. I will edit my original post to explain what x42 is and why its distribution is like this. $\endgroup$ – hooyeh Feb 15 '17 at 19:04
1
$\begingroup$

OP's question has an answer and that is by bootstrapping the regression problem and examining the confidence intervals of parameters and fit.

For example, SPSS can do MLR bootstrap confidence intervals. This cannot be done here as the OP did not supply any data, divulge the regression model, or supply any testing information on selection of regression methods such as why one should be using the OLS regression method.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.