Linear Regression: How can the error term have variance?

Question

I seem to be missing something fundamental about the structure of the linear regression model.

Suppose we have a response variable $Y$ and $p$ predictor variables $X_1$ to $X_p$.

For a particular sample of sample size $n$ we seek to model $Y$ in terms of the predictor variables so that for the $i$-th observation we get

$y_i = \beta_0+\beta_1x_{i1}+\beta_2x_{i2} +...+ \beta_px_{ip}+\epsilon_i$

So we deal with $n$ error terms $\epsilon_1$ to $\epsilon_n$ - $n$ numbers, one for each observation. What does it mean, in the context of least squares estimation, to stipulate that each of the $\epsilon_i$ be homoscedascic, i.e. each of them has the same variance $\sigma^2$ when all we have is one error term for each observation? How can we talk about variance of $\epsilon_i$ when it is a single number rather than a random variable which has a distribution and hence variance?

Answer 1

To attempt an answer to your original question, each $\epsilon_i$ is considered to be one realization of the random variable $\epsilon$. So each $\epsilon_i$, $i = 1,\ldots,n$, is indeed only one number, but there is an underlying distribution which generates those numbers. I have never seen the notion of homo/heteroscedasticity as referring to any single one $\epsilon_i$ in particular (and would certainly not choose that phrasing myself), rather the concept refers to the whole collection of error terms.

A simple example would be to consider that you are taking 50 measurements manually, using some tool that requires hand stability, so your sample is $y_1,\ldots,y_{50}$. Then a friend of yours with shaky hands takes another 50 measurements, $y_{51},\ldots,y_{100}$. Intuitively, if you model your 50 observations separately from those of your friend, you'd expect that your $\epsilon_i, i=1,\ldots,50,$ would tend to deviate less from their mean value than those $\epsilon_i, i=51,\ldots,100$ your friend would get from his model on the same predictors, because you know you have steadier hands. This means that your $\epsilon_i$ are in fact homoscedastic within their own group, i.e. for $i=1,\ldots,50$ (assuming you don't get tired after a while). What's more, your friend's errors are also homoscedastic with respect to his group, i.e. $i=51,\ldots,100$. If you pool everything together though, the collection $\epsilon_i, i=1,\ldots,100$ is decidedly not homoscedastic, because not all of these $\epsilon_i$ were generated by the same mechanism (half the errors came from your hands, the other half from your shaky friend).

Another example of heteroscedasticity would be to take all 100 measurements yourself but (much more realistically) assume fatigue with time: you'd see that your $\epsilon_i$ would be very low for the measurements you took first, but they would increase with time as your muscles grow tired and your precision suffers, thus increasing the variance of future errors (i.e. the chance that you make bigger errors in the measurement). Hope this helps.

Answer 2

It's easier to understand on a concrete example. Suppose, you want to predict the precipitation based on the temperature of a site. So, you posit a model: $$y_i=\beta_0+\beta x_i+\varepsilon_i$$ Here, $y_i$ average precipitation and $x_i$ - average temperature in town. Do you think that the prediction uncertainty is the same regardless of a town? If you do, then you assume that the errors are homoscedastic.

However, one could make an argument that the uncertainty is not the same for all geographies. Maybe for some reason this model forecasts better for Ulanbaator than for Washington DC. In this case the error variances will be larger for Washington, and smaller for Ulanbaator, i.e. heteroscedastic errors

I suspect that you naturally consider the errors $\varepsilon_i$ as realizations of some random variable $\varepsilon$. Hence, you think of a variance $\sigma^2$ as a variance of that variable $Var[\varepsilon]$. It seems to fit the intuition. However, in reality each error terms $\varepsilon_i$ is a random variable itself. Each one of these have their own variance $Var[\varepsilon_i]=\sigma_i^2$, they don't need to be the same