3
$\begingroup$

I seem to be missing something fundamental about the structure of the linear regression model.

Suppose we have a response variable $Y$ and $p$ predictor variables $X_1$ to $X_p$.

For a particular sample of sample size $n$ we seek to model $Y$ in terms of the predictor variables so that for the $i$-th observation we get

$y_i = \beta_0+\beta_1x_{i1}+\beta_2x_{i2} +...+ \beta_px_{ip}+\epsilon_i$

So we deal with $n$ error terms $\epsilon_1$ to $\epsilon_n$ - $n$ numbers, one for each observation. What does it mean, in the context of least squares estimation, to stipulate that each of the $\epsilon_i$ be homoscedascic, i.e. each of them has the same variance $\sigma^2$ when all we have is one error term for each observation? How can we talk about variance of $\epsilon_i$ when it is a single number rather than a random variable which has a distribution and hence variance?

$\endgroup$
  • 1
    $\begingroup$ Hi, welcome. Homoskedasticity means there is one $\sigma$ for all $\varepsilon_i$. Heteroskedasticity would be implied by an error variance that is a function of the regressors $\mathbf{X}$. For a simple example of heteroskedasticity see the second image in stats.meta.stackexchange.com/a/613 where $\text{Speed}$ is regressed on $\text{Rep}$. $\endgroup$ – Jim Jul 17 '18 at 17:06
  • $\begingroup$ How could there be more than one $\sigma$, more than one variance measure for the set of $n$ numbers $\epsilon_i$? I don't get how for the continuous range of $X$ the condition of equal variances is defined. Equal to what? Is the idea to say that for any fixed subrange or interval of $X$ the error terms of the model must have the same variance for homoscedasticity to hold? $\endgroup$ – Redstart Jul 17 '18 at 18:35
  • $\begingroup$ A practical example of having more than one $\sigma$ is a posttest-only two-group randomized controlled study where the treatment has a homogenizing effect. In this example, $\epsilon$ is actually composed of two random variables, with different distributions, both mean zero, but larger $\sigma$ for the control than for the treatment. $\endgroup$ – Heteroskedastic Jim Jul 18 '18 at 12:49
3
$\begingroup$

To attempt an answer to your original question, each $\epsilon_i$ is considered to be one realization of the random variable $\epsilon$. So each $\epsilon_i$, $i = 1,\ldots,n$, is indeed only one number, but there is an underlying distribution which generates those numbers. I have never seen the notion of homo/heteroscedasticity as referring to any single one $\epsilon_i$ in particular (and would certainly not choose that phrasing myself), rather the concept refers to the whole collection of error terms.

A simple example would be to consider that you are taking 50 measurements manually, using some tool that requires hand stability, so your sample is $y_1,\ldots,y_{50}$. Then a friend of yours with shaky hands takes another 50 measurements, $y_{51},\ldots,y_{100}$. Intuitively, if you model your 50 observations separately from those of your friend, you'd expect that your $\epsilon_i, i=1,\ldots,50,$ would tend to deviate less from their mean value than those $\epsilon_i, i=51,\ldots,100$ your friend would get from his model on the same predictors, because you know you have steadier hands. This means that your $\epsilon_i$ are in fact homoscedastic within their own group, i.e. for $i=1,\ldots,50$ (assuming you don't get tired after a while). What's more, your friend's errors are also homoscedastic with respect to his group, i.e. $i=51,\ldots,100$. If you pool everything together though, the collection $\epsilon_i, i=1,\ldots,100$ is decidedly not homoscedastic, because not all of these $\epsilon_i$ were generated by the same mechanism (half the errors came from your hands, the other half from your shaky friend).

Another example of heteroscedasticity would be to take all 100 measurements yourself but (much more realistically) assume fatigue with time: you'd see that your $\epsilon_i$ would be very low for the measurements you took first, but they would increase with time as your muscles grow tired and your precision suffers, thus increasing the variance of future errors (i.e. the chance that you make bigger errors in the measurement). Hope this helps.

$\endgroup$
  • 1
    $\begingroup$ I love this example. It clearly demonstrates the error term as a realization of a random variable. $\endgroup$ – Noah Jul 18 '18 at 20:28
  • $\begingroup$ Very happy to hear that. Hopefully others also found the answer useful, though from what I read in the question, OP had an issue with the definition of homoscedasticity in the sense that it's not defined w.r.t to a single observation but rather by looking at any slice & dice (i.e. random sub-samples) combination of the data, so if all these combinations have the same variance, then we have homoscedasticity. $\endgroup$ – Emil Jul 18 '18 at 22:20
1
$\begingroup$

It's easier to understand on a concrete example. Suppose, you want to predict the precipitation based on the temperature of a site. So, you posit a model: $$y_i=\beta_0+\beta x_i+\varepsilon_i$$ Here, $y_i$ average precipitation and $x_i$ - average temperature in town. Do you think that the prediction uncertainty is the same regardless of a town? If you do, then you assume that the errors are homoscedastic.

However, one could make an argument that the uncertainty is not the same for all geographies. Maybe for some reason this model forecasts better for Ulanbaator than for Washington DC. In this case the error variances will be larger for Washington, and smaller for Ulanbaator, i.e. heteroscedastic errors

I suspect that you naturally consider the errors $\varepsilon_i$ as realizations of some random variable $\varepsilon$. Hence, you think of a variance $\sigma^2$ as a variance of that variable $Var[\varepsilon]$. It seems to fit the intuition. However, in reality each error terms $\varepsilon_i$ is a random variable itself. Each one of these have their own variance $Var[\varepsilon_i]=\sigma_i^2$, they don't need to be the same

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.