8
$\begingroup$

I want to perform linear regression on some data. For every value of x, the data values are distributed normally across y, around some mean. However, the variance increases linearly as x increases. I made this example graph:

linear regression graph

Blue is the regression line, red are data points, black shows the normal distribution, and green visualizes the variance increasing.

How can I calculate a regression for the change in variance, while also performing a linear regression of the data? The data is heteroscedastic, and I've read up on methods for doing linear regression on such data. However, I haven't found anything on estimating the actual change in variance of the data.

I haven't studied stats rigorously, so any simple explanations or resources I could look at further would be appreciated.

More Details:

The original dataset follows $y = a/x + b$. The variance as $x$ changes follows a similar model $s^2 = c/x + d$. I transformed the data using $x' = 1/x$ to make the data linear (just to simplify the problem). Here is a sample graph (left is transformed, right is original):

enter image description here

$\endgroup$
15
  • $\begingroup$ Possible duplicate of What does having "constant variance" in a linear regression model mean? $\endgroup$
    – gammer
    Apr 26, 2017 at 5:16
  • 1
    $\begingroup$ Do those lines meet (ie the place where the implied variance is 0) at $x=0$? Or somewhere else? If somewhere else is its x-value known or unknown? What makes you assert that the distribution is normal? Is that a guess? $\endgroup$
    – Glen_b
    Apr 26, 2017 at 5:21
  • 1
    $\begingroup$ It depends; there are other formulations that may suit your data better, but the normal is fine if that's what you want. Are the data always $\geq 0$? would $E(Y|X=0) =0$ or not? $\endgroup$
    – Glen_b
    Apr 26, 2017 at 5:41
  • 1
    $\begingroup$ @Matthew I'm still trying to find out the exact problem we're solving. In some cases there are several interesting options with easy implementations; in other cases there are fewer options. $\endgroup$
    – Glen_b
    Apr 26, 2017 at 5:52
  • 2
    $\begingroup$ See stats.stackexchange.com/questions/258485/… for some ideas $\endgroup$ Apr 26, 2017 at 12:25

3 Answers 3

6
$\begingroup$

This sounds like a special case of heteroscedasticity.

There are two issues:

  1. What estimator should you use in the presence of heteroscedasticity?
  2. How should you calculate your standard errors?

The most straightforward thing to do is run a regular regression but use heteroscedastic robust standard errors. As @Glen_b suggests in the comments though, you probably can do better than this by efficiently exploiting known structure on your problem.

What estimator to use?

  • You could just run a normal regression.

  • You could run weighted least squares, an application of generalized least squares. The loose idea is to give more weight to observations with low variance error terms.

    • Since you probably don't know ex-ante how the variance of the error term varies with $x$, you probably have to do something like feasible gls.

If you run a regular OLS regression, you should not use the usual standard errors based upon assumptions of homoscedasticity. Instead you should use heteroscedastic robust standard errors. Any stats package can do this.

$\endgroup$
1
  • 1
    $\begingroup$ I don't think this answers the question. The question is not just about estimating the errors, but about estimating the change of the variance of those errors, as a function of x. I think in this case the OP is assuming that that function will be linear. $\endgroup$
    – naught101
    Oct 1, 2020 at 0:06
3
$\begingroup$

Your data violate the assumption of homoscedasticity. You can use a regression method that produces standard errors that are robust to heteroscedasticity. What software are you using to run your regression? If you are using R, you can use the sandwich package to estimate robust standard errors.

$\endgroup$
8
  • $\begingroup$ Won't generic variance formulations throw out almost all the information about variance specified in the question? $\endgroup$
    – Glen_b
    Apr 26, 2017 at 5:23
  • $\begingroup$ I'm not sure what you mean—could you elaborate? $\endgroup$
    – Mark White
    Apr 26, 2017 at 5:27
  • 1
    $\begingroup$ I would like to get a measure for how the variance is changing as well, in addition to the linear regression. $\endgroup$
    – Azmisov
    Apr 26, 2017 at 5:29
  • 1
    $\begingroup$ @Mark Unless I misunderstand what you're suggesting, the usual sandwich estimator estimates the variance pointwise (using squared residuals to estimate variances); it's good in that it's consistent and doesn't need to rely on any assumption about the form of the heteroskedasticity, so it's worth raising as an option at least. However the OP has specified a complete model for the variance (up to a single parameter - essentially the variance at unit distance from wherever the variance is 0). You're tossing out a great deal of information that the OP treats as known, aren't you? Or am I confused? $\endgroup$
    – Glen_b
    Apr 26, 2017 at 5:37
  • 2
    $\begingroup$ Actually, it (a variance-model) is not at all uncommon in science and a few other application areas (though perhaps rarer in the disciplines that tend to use the sandwich-type estimators) $\endgroup$
    – Glen_b
    Apr 26, 2017 at 6:05
1
$\begingroup$

Machine learning guy here -- as much as I love the stats folks ML usually wins the day in real-world applications, and heteroscedasticity is one such common occurrence. A more general solution that is non-parametric and works for this and other (even highly) nonlinear regression problems is to just use quantile regression with a bagged decision tree. You can follow some links at Matlab to learn more (their examples are easily transferable to Python or R).

$\endgroup$
1
  • 1
    $\begingroup$ This is not so useful if you only have a small sample size, methinks. But definitely a good option if you have a lot of data. $\endgroup$
    – naught101
    Oct 1, 2020 at 0:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.