I want to perform linear regression on some data. For every value of x, the data values are distributed normally across y, around some mean. However, the variance increases linearly as x increases. I made this example graph:

linear regression graph

Blue is the regression line, red are data points, black shows the normal distribution, and green visualizes the variance increasing.

How can I calculate a regression for the change in variance, while also performing a linear regression of the data? The data is heteroscedastic, and I've read up on methods for doing linear regression on such data. However, I haven't found anything on estimating the actual change in variance of the data.

I haven't studied stats rigorously, so any simple explanations or resources I could look at further would be appreciated.

More Details:

The original dataset follows $y = a/x + b$. The variance as $x$ changes follows a similar model $s^2 = c/x + d$. I transformed the data using $x' = 1/x$ to make the data linear (just to simplify the problem). Here is a sample graph (left is transformed, right is original):

enter image description here

  • $\begingroup$ Possible duplicate of What does having "constant variance" in a linear regression model mean? $\endgroup$ – gammer Apr 26 '17 at 5:16
  • 1
    $\begingroup$ Do those lines meet (ie the place where the implied variance is 0) at $x=0$? Or somewhere else? If somewhere else is its x-value known or unknown? What makes you assert that the distribution is normal? Is that a guess? $\endgroup$ – Glen_b Apr 26 '17 at 5:21
  • 1
    $\begingroup$ It depends; there are other formulations that may suit your data better, but the normal is fine if that's what you want. Are the data always $\geq 0$? would $E(Y|X=0) =0$ or not? $\endgroup$ – Glen_b Apr 26 '17 at 5:41
  • 1
    $\begingroup$ @Matthew I'm still trying to find out the exact problem we're solving. In some cases there are several interesting options with easy implementations; in other cases there are fewer options. $\endgroup$ – Glen_b Apr 26 '17 at 5:52
  • 2
    $\begingroup$ See stats.stackexchange.com/questions/258485/… for some ideas $\endgroup$ – kjetil b halvorsen Apr 26 '17 at 12:25

This sounds like a special case of heteroscedasticity.

There are two issues:

  1. What estimator should you use in the presence of heteroscedasticity?
  2. How should you calculate your standard errors?

The most straightforward thing to do is run a regular regression but use heteroscedastic robust standard errors. As @Glen_b suggests in the comments though, you probably can do better than this by efficiently exploiting known structure on your problem.

What estimator to use?

  • You could just run a normal regression.

  • You could run weighted least squares, an application of generalized least squares. The loose idea is to give more weight to observations with low variance error terms.

    • Since you probably don't know ex-ante how the variance of the error term varies with $x$, you probably have to do something like feasible gls.

If you run a regular OLS regression, you should not use the usual standard errors based upon assumptions of homoscedasticity. Instead you should use heteroscedastic robust standard errors. Any stats package can do this.

| cite | improve this answer | |

Your data violate the assumption of homoscedasticity. You can use a regression method that produces standard errors that are robust to heteroscedasticity. What software are you using to run your regression? If you are using R, you can use the sandwich package to estimate robust standard errors.

| cite | improve this answer | |
  • $\begingroup$ Won't generic variance formulations throw out almost all the information about variance specified in the question? $\endgroup$ – Glen_b Apr 26 '17 at 5:23
  • $\begingroup$ I'm not sure what you mean—could you elaborate? $\endgroup$ – Mark White Apr 26 '17 at 5:27
  • $\begingroup$ I would like to get a measure for how the variance is changing as well, in addition to the linear regression. $\endgroup$ – Azmisov Apr 26 '17 at 5:29
  • 1
    $\begingroup$ @Mark Unless I misunderstand what you're suggesting, the usual sandwich estimator estimates the variance pointwise (using squared residuals to estimate variances); it's good in that it's consistent and doesn't need to rely on any assumption about the form of the heteroskedasticity, so it's worth raising as an option at least. However the OP has specified a complete model for the variance (up to a single parameter - essentially the variance at unit distance from wherever the variance is 0). You're tossing out a great deal of information that the OP treats as known, aren't you? Or am I confused? $\endgroup$ – Glen_b Apr 26 '17 at 5:37
  • 1
    $\begingroup$ Actually, it (a variance-model) is not at all uncommon in science and a few other application areas (though perhaps rarer in the disciplines that tend to use the sandwich-type estimators) $\endgroup$ – Glen_b Apr 26 '17 at 6:05

Machine learning guy here -- as much as I love the stats folks ML usually wins the day in real-world applications, and heteroscedasticity is one such common occurrence. A more general solution that is non-parametric and works for this and other (even highly) nonlinear regression problems is to just use quantile regression with a bagged decision tree. You can follow some links at Matlab to learn more (their examples are easily transferable to Python or R).

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.