How to conceptualize error in a regression model?

Question

I am attending a data analysis class and some of my well-rooted ideas are being shaken. Namely, the idea that the error (epsilon), as well as any other sort of variance, applies only (so I thought) to a group (a sample or whole population). Now, we're being taught that one of the regression assumptions is that the variance is "the same for all individuals". This is somehow shocking to me. I always thought that it was the variance in Y accross all values of X that was assumed to be constant.

I had a chat with the prof, who told me that when we do a regression, we assume our model to be true. And I think that's the tricky part. To me, the error term (epsilon) always meant something like "whatever elements we don't know and that might affect our outcome variable, plus some measurement error". In the way the class is taught, there's no such thing as "other stuff"; our model is assumed to be true and complete. This means that all residual variation has to be thought of as a product of measurement error (thus, measuring an individual 20 times would be expected to produce the same variance as measuring 20 individuals one time).

I feel something's wrong somewhere, I'd like to have some expert opinion on this... Is there some room for interpretation as to what the error term is, conceptually speaking?

Answer 1

If there are aspects of individuals that have an effect on the resulting y values, then either there is some way at getting at those aspects (in which case they should be part of the predictor x), or there's no way of ever getting at that information.

If there's no way of ever getting at this information and there's no way of repeatedly measuring y values for individuals, then it really doesn't matter. If you can measure y repeatedly, and if your data set actually contains repeated measurements for some individuals, then you've got a potential problem on your hands, since the statistical theory assumes independence of the measurement errors/residuals.

For example, suppose that you're trying to fit a model of the form

$y=\beta_0+\beta_1 x$,

and that for each individual,

$yind=100+10x+z$,

where z depends on the individual and is normally distributed with mean 0 and standard deviation 10. For each repeated measurement of an individual,

$ymeas=100+10x+z+e$,

where $e$ is normally distributed with mean 0 and standard deviation 0.1.

You could try to model this as

$y=\beta_0+\beta_1 x+\epsilon$,

where $\epsilon$ is normally distributed with mean 0 and standard deviation

$\sigma=\sqrt{10^2+0.1^2}=\sqrt{100.01}$.

As long as you only have one measurement for each individual, that would be fine. However, if you have multiple measurements for the same individual, then your residuals will no longer be independent!

For example, if you have one individual with z=15 (1.5 standard deviations out, so not that unreasonable), and one hundred repeated measurements of that individual, then using $\beta_0=100$ and $\beta_1=10$ (the exact values!) you'd end up with 100 residuals of about +1.5 standard deviations, which would look extremely unlikely. This would effect the $\chi^2$ statistic.

Answer 2

I think "error" is best described as "the part of the observations that is unpredtictable given our current information". Trying to think in terms of population vs sample leads to conceptual problems (well it does for me anyway), as does thinking of the errors as "purely random" drawn from some distribution. thinking in terms of prediction and "predictability" makes much more sense to me.

I also think the maximum entropy principle provides a neat way to understand why a normal distribution is used. For when modelling we are assigning a distribution to the errors to describe what is known about them. Any joint distribution $p(e_{1},\dots,e_{n})$ could represent a conceivable state of knowledge. However if we specify some structure such as $E(\frac{1}{n}\sum_{i=1}^{n}e_{i}^2)=\sigma^2$ then the most uniform distribution subject to this constraint is the normal distribution with zero mean and constant variance $\sigma^2$. This shows that "independence" and "constant variance" are actually safer than assuming otherwise under this constraint - namely that the average second moment exists and is finite and we expect the general size of the errors to be $\sigma$.

So one way to think of this is that we do not necessarily think our assumptions are "correct" but rather "safe" in the sense that we are not injecting a lot of information into the problem (we are imposing just one structural constraint in $n$ dimensions). so we are starting from a safe area - and we can build up from here depending on what specific information we have about the particular case and data set at hand.

Answer 3

Here is very useful link to explain simple linear regression : http://www.dangoldstein.com/dsn/archives/2006/03/every_wonder_ho.html maybe it can help to grasp the "error" concept.

F.D.

Answer 4

I disagree with the professor's formulation of this. As you say, the idea that the variance is the same for each individual implies that the error term represents only measurement error. This is not usually how the basic multiple regression model is constructed. Also as you say, variance is defined for a group (whether it's a group of individual subjects or a group of measurements). It doesn't apply at the individual level, unless you have repeated measures.

A model needs to be complete in that the error term should not contain influences from any variables that are correlated with predictors. The assumption is that the error term is independent of predictors. If some correlated variable is omitted, you will get biased coefficients (this is called omitted variable bias).