# How to conceptualize error in a regression model?

I am attending a data analysis class and some of my well-rooted ideas are being shaken. Namely, the idea that the error (epsilon), as well as any other sort of variance, applies only (so I thought) to a group (a sample or whole population). Now, we're being taught that one of the regression assumptions is that the variance is "the same for all individuals". This is somehow shocking to me. I always thought that it was the variance in Y accross all values of X that was assumed to be constant.

I had a chat with the prof, who told me that when we do a regression, we assume our model to be true. And I think that's the tricky part. To me, the error term (epsilon) always meant something like "whatever elements we don't know and that might affect our outcome variable, plus some measurement error". In the way the class is taught, there's no such thing as "other stuff"; our model is assumed to be true and complete. This means that all residual variation has to be thought of as a product of measurement error (thus, measuring an individual 20 times would be expected to produce the same variance as measuring 20 individuals one time).

I feel something's wrong somewhere, I'd like to have some expert opinion on this... Is there some room for interpretation as to what the error term is, conceptually speaking?

-
Perhaps what he meant was that, even if the model is true, there is still random variation in the responses - this is captured by the error variance - this can, for example, be attributed to an imperfect measurement apparatus. Others sometimes conceptualize the error variance as that due to missing predictors (not necessarily errors in the form of the model), implying that if all possible predictors were measured, the error variance would be 0. This is not inconsistent with the first - the errors in measurement can be thought of as a "missing predictor". –  Macro Feb 8 '12 at 1:48
I think one thing that's always hard to grasp at first is that "error" could mean different things in this instance. "Error" could refer to the difference between the fitted values we obtain from our model and the observed values (the discrepancy can be due to a fairly parsimonious model, e.g.). "Error" could also mean the difference between the observed values and the true values (the discrepancy can be due to, say, the device you use to measure the values rounds to the nearest integer/tenth decimal/etc.). [The first type is where you'd hear terms like "residuals/residual variance."] –  Mike Wierzbicki Feb 8 '12 at 3:39
@Macro Yes, this seems to me like a natural way of thinking of error. I'm trying however to understand why the prof insisted on the stricter definition of it (thinking of it as applicable to each individual even though we know in reality, it isn't true). –  dominic999 Feb 8 '12 at 4:01
@MikeWierzbicki Right. And if I understand correctly, this is all lumped together in the "strict" viewpoint. Meaning that all the difference between observed and predicted values comes from measurement error, since our model "has to be true". –  dominic999 Feb 8 '12 at 4:07

I disagree with the professor's formulation of this. As you say, the idea that the variance is the same for each individual implies that the error term represents only measurement error. This is not usually how the basic multiple regression model is constructed. Also as you say, variance is defined for a group (whether it's a group of individual subjects or a group of measurements). It doesn't apply at the individual level, unless you have repeated measures.

A model needs to be complete in that the error term should not contain influences from any variables that are correlated with predictors. The assumption is that the error term is independent of predictors. If some correlated variable is omitted, you will get biased coefficients (this is called omitted variable bias).

-
I do not quite understand this answer. it appears to recognize the difference between error due to lack of fit and random error, but the last rhetorical question seems confusing. From a purely formal perspective, essentially any inference made with respect to a regression model hinges on very explicit assumptions about the noise structure. –  cardinal Feb 8 '12 at 13:52
My point is that in many cases, the purpose of regression modeling is to figure out what's happening even when we don't know all the causes of a particular outcome. But as it seems unclear, I will remove that question. –  Anne Z. Feb 8 '12 at 14:57
Thanks. The point in your comment is good. The previous question you stated could be read as questioning the entire basis on which regression theory rests. :) –  cardinal Feb 8 '12 at 15:10
I agree with you in your disagreement (hence my question!), and the omitted variable bias is quite relevant to the issue. Thanks. –  dominic999 Feb 8 '12 at 17:43

I think "error" is best described as "the part of the observations that is unpredtictable given our current information". Trying to think in terms of population vs sample leads to conceptual problems (well it does for me anyway), as does thinking of the errors as "purely random" drawn from some distribution. thinking in terms of prediction and "predictability" makes much more sense to me.

I also think the maximum entropy principle provides a neat way to understand why a normal distribution is used. For when modelling we are assigning a distribution to the errors to describe what is known about them. Any joint distribution $p(e_{1},\dots,e_{n})$ could represent a conceivable state of knowledge. However if we specify some structure such as $E(\frac{1}{n}\sum_{i=1}^{n}e_{i}^2)=\sigma^2$ then the most uniform distribution subject to this constraint is the normal distribution with zero mean and constant variance $\sigma^2$. This shows that "independence" and "constant variance" are actually safer than assuming otherwise under this constraint - namely that the average second moment exists and is finite and we expect the general size of the errors to be $\sigma$.

So one way to think of this is that we do not necessarily think our assumptions are "correct" but rather "safe" in the sense that we are not injecting a lot of information into the problem (we are imposing just one structural constraint in $n$ dimensions). so we are starting from a safe area - and we can build up from here depending on what specific information we have about the particular case and data set at hand.

-
What do you mean by 'uniform' in: "then the most uniform distribution subject to this constraint is the normal distribution with zero mean and constant variance $\sigma^2$"? –  Macro Feb 8 '12 at 3:28
I mean $p(e_{1},\dots,e_{n})\propto 1$ ie a uniform distribution. –  probabilityislogic Feb 8 '12 at 4:42
And by close i mean kl divergence is minimised –  probabilityislogic Feb 8 '12 at 4:47
The dilemma is not between sample and population. It's about thinking of error as applicable to individuals vs to the sample/population. –  dominic999 Feb 8 '12 at 17:34

If there are aspects of individuals that have an effect on the resulting y values, then either there is some way at getting at those aspects (in which case they should be part of the predictor x), or there's no way of ever getting at that information.

If there's no way of ever getting at this information and there's no way of repeatedly measuring y values for individuals, then it really doesn't matter. If you can measure y repeatedly, and if your data set actually contains repeated measurements for some individuals, then you've got a potential problem on your hands, since the statistical theory assumes independence of the measurement errors/residuals.

For example, suppose that you're trying to fit a model of the form

$y=\beta_0+\beta_1 x$,

and that for each individual,

$yind=100+10x+z$,

where z depends on the individual and is normally distributed with mean 0 and standard deviation 10. For each repeated measurement of an individual,

$ymeas=100+10x+z+e$,

where $e$ is normally distributed with mean 0 and standard deviation 0.1.

You could try to model this as

$y=\beta_0+\beta_1 x+\epsilon$,

where $\epsilon$ is normally distributed with mean 0 and standard deviation

$\sigma=\sqrt{10^2+0.1^2}=\sqrt{100.01}$.

As long as you only have one measurement for each individual, that would be fine. However, if you have multiple measurements for the same individual, then your residuals will no longer be independent!

For example, if you have one individual with z=15 (1.5 standard deviations out, so not that unreasonable), and one hundred repeated measurements of that individual, then using $\beta_0=100$ and $\beta_1=10$ (the exact values!) you'd end up with 100 residuals of about +1.5 standard deviations, which would look extremely unlikely. This would effect the $\chi^2$ statistic.

-
I tried to avoid using the scary term "multilevel modeling" in my answer, but you should be aware that in some cases it provides a way to deal with this kind of situation. –  Brian Borchers Feb 8 '12 at 3:47