Is the linearity assumption in linear regression merely a definition of $\epsilon$?

Question

I am revising linear regression.

The textbook by Greene states:

Now, of course there will be other assumptions on the linear regression model, such as $E(\epsilon|X)=0$. This assumption combined with the linearity assumption (which in effect defines $\epsilon$), puts structure on the model.

However, the linearity assumption by itself does not put any structure on our model, since $\epsilon$ can be completely arbitrary. For any variables $X, y$ whatsoever, no matter what the relation between the two we could define an $\epsilon$ such that the linearity assumption holds. Therefore, the linearity "assumption" should really be called a definition of $\epsilon$, rather than an assumption.

Therefore I am wondering:

1. Is Greene being sloppy? Should he actually have written: $E(y|X)=X\beta$? This is a "linearity assumption" that actually puts structure on the model.

2. Or do I have to accept that the linearity assumption does not put structure on the model but only defines an $\epsilon$, where the other assumptions will use that definition of $\epsilon$ to put structure on the model?

---

Edit: since there seems to be some confusion around the other assumptions, let me add the full set of assumptions here:

This is from Greene, Econometric Analysis, 7th ed. p. 16.

Answer 1

1. Is Greene being sloppy? Should he actually have written: $E(y|X)=X\beta$? This is a "linearity assumption" that actually puts structure on the model.

In a sense, yes and no. In the one hand, yes, given current modern causality research he is sloppy, but just like most econometrics textbooks are, in the sense that they do not make a clear distinction of causal and observational quantities, leading to common confusions like this very question. But, in the other hand, no, this assumption is not sloppy in the sense that it is indeed different from simply assuming $E(y|X)=X\beta$.

The crux of the matter here is the difference between the conditional expectation, $E(y|X)$, and the structural (causal) equation of $y$, as well as its structural (causal) expectation $E[Y|do(X)]$. The linearity assumption in Greene is a structural assumption. Let's see a simple example. Imagine the structural equation is:

$$ y= \beta x + \gamma x^2 + \epsilon $$

Now let $E[\epsilon |x] = \delta x - \gamma x^2$. Then we would have:

$$ E[y|x] = \beta'x $$

where $\beta' = \beta + \delta$. Moreover, we can write $y = \beta'x + \epsilon'$ and we would have $E[\epsilon'|x] = 0$. This shows we can have a correctly specified linear conditional expectation $E[y|x]$ which by definition is going to have an orthogonal disturbance, yet the structural equation would be nonlinear.

2. Or do I have to accept that the linearity assumption does not put structure on the model but only defines an $\epsilon$, where the other assumptions will use that definition of $\epsilon$ to put structure on the model?

The linearity assumption does define an $\epsilon$, that is, $\epsilon := y - X\beta = y - E[Y|do(X)]$ by definition, where $\epsilon$ represents the deviations of $y$ from its expectation when we experimentally set $X$ (see Pearl section 5.4). The other assumptions are used either for identification of the structural parameters (for instance, the assumption of exogeneity of $\epsilon$ allows you to identify the structural expectation $E[Y|do(X)]$ with the conditional expectation $E[Y|X]$) or for derivation of statistical properties of the estimators (for instance, the assumption of homoskedasticity guarantees OLS is BLUE, the assumption of normality makes it easy to derive "finite sample" results for inference etc).

However, the linearity assumption by itself does not put any structure on our model, since $\epsilon$ can be completely arbitrary. For any variables $X, y$ whatsoever, no matter what the relation between the two we could define an $\epsilon$ such that the linearity assumption holds.

Your statement here goes into the main problem of causal inference in general! As shown in the simple example above, we can cook up structural disturbances that could make the conditional expectation of $y$ given $x$ linear. In general, several different structural (causal) models can have the same observational distribution, you can even have causation without observed association. Therefore, in this sense, you are correct --- we need more assumptions on $\epsilon$ in order to put "more structure" into the problem and identify the structural parameters $\beta$ with observational data.

Side note

It's worth mentioning most econometrics textbooks are confusing when it comes to the distinction between regression and structural equations and their meaning. This has been documented lately. You can check a paper by Chen and Pearl here as well as an extended survey by Chris Auld. Greene is one of the books examined.

Answer 2

edited after comments by OP and Matthew Drury

To answer this question I assume Greene, and OP, have the following definition of linearity in mind: Linearity means that for every one unit increase in this predictor, the outcome is increased by beta ($β$), wherever on the range of possible predictor values this one-unit increase occurs. I.e. the function $y=f(x)$ is $y=a+bx$ and not e.g. $y=a+bx^2$ or $y=a+sin(x)$. Further, this assumption is focused on the betas and thus applies to predictors (aka independent variables).

The expectation of residuals conditional on the model $E(ϵ|X)$ is something else. Yes, it is true that the math behind a linear regression defines/tries to define $E(ϵ|X)=0$. However, this is usually set over the entire range of fitted/predicted values for $y$. If you look at specific parts of the linear predictor and the predicted value of $y$, you might notice heteroscedasticity (areas where the variation of $ϵ$ is larger than elsewhere), or areas where $E(ϵ|X)≠0$. A non-linear association between the $x$'s and $y$ might be the cause for this, but is not the only reason heteroscedasticity or $E(ϵ|X)≠0$ might occur (see for example missing predictor bias).

From the comments: OP states "the linearity assumption does not restrict the model in any way, given that epsilon is arbitrary and can be any function of XX whatsoever", to which I would agree. I think this is made clear by linear regressions being able to fit to any data, whether or not the linearity assumption is violated or not. I'm speculating here, but that might be the reason why Greene chose to keep the error $ϵ$ in the formula - saving the $E(ϵ|X)=0$ for later - to denote that in assuming linearity, $y$ (and not the expected $y$) can be defined based on $X$ but maintains some error $ϵ$, regardless of what values $ϵ$ takes. I can only hope that he would later go on to state the relevancy of $E(ϵ|X)=0$.

In short (admittedly, without fully reading Greene's book and checking his argumentation):

1. Greene probably refers to the betas being constant for the entire range of the predictor (emphasis should be placed on the beta in the $y=Xβ + ϵ$ or $E(ϵ|X)=Xβ$ equations;
2. The linearity assumption does put some structure on the model. You should however note that transformations or additions such as splines prior to modelling, can make non-linear associations conform to the linear regression framework.

Answer 3

Suppose we're not considering causality at all and just viewing linear regression as a statistical model. In that case, the OP is right that Assumption A1 is trivial and that the real assumption comes in with A3. However, if we aren't considering causality at all, then it's misleading to call assumption A3 "exogeneity," which is a casual concept. Instead, you can just call it "mean independence of $\varepsilon$ and $x$."

For more, see this blog post: https://evangorstein.github.io/2024/07/04/linear-regression/

Answer 4

I was a little confused by the answer above, hence I'll give it another shot. I think the question is not actually about 'classical' linear regression but about the style of that particular source. On the classical regression part:

However, the linearity assumption by itself does not put any structure on our model

That is absolutely correct. As you have stated, $\epsilon$ might as well kill the linear relation and add up something completely independent from $X$ so that we cannot compute any model at all.

Is Greene being sloppy? Should he actually have written: $E(y|X)=Xβ$

I do not want to answer the first question but let me sum up the assumptions you need for usual linear regression:

Let us assume that you observe (you are given) data points $x_i \in \mathbb{R}^d$ and $y_i \in \mathbb{R}$ for $i=1,...,n$. You need to assume that the data $(x_i, y_i)$ you have observed comes from independently, identically distributed random variables $(X_i, Y_i)$ such that ...

1. There exists a fixed (independent of $i$) $\beta \in \mathbb{R}^d$ such that $Y_i = \beta X_i + \epsilon_i$ for all $i$ and the random variables $\epsilon_i$ are such that

2. The $\epsilon_i$ are iid as well and $\epsilon_i$ is distributed as $\mathcal{N}(0, \sigma)$ ($\sigma$ must be independent of $i$ as well)

3. For $X = (X_1, ..., X_n)$ and $Y = (Y_1, ..., Y_n)$ the variables $X, Y$ have a common density, i.e. the single random variable $(X, Y)$ has a density $f_{X,Y}$

Now you can run down the usual path and compute

$$f_{Y|X}(y|x) = f_{Y,X}(y,x)/f_X(x) = \left(\frac{1}{\sqrt{2\pi d}}\right)^n \exp{\left( \frac{-\sum_{i=1}^n (y_i - \beta x_i)^2}{2\sigma}\right)} $$

so that by the usual 'duality' between machine learning (minimalization of error functions) and probability theory (maximization of likelihoods) you maximize $-\log f_{Y|X}(y|x)$ in $\beta$ which in fact, gives you the usual "RMSE" stuff.

Now as stated: If the author of the book you are quoting wants to make this point (which you have to do if you ever want to be able to compute the 'best possible' regression line in the basic setup) then yes, he must make this assumption on the normalicity of the $\epsilon$ somewhere in the book.

There are different possibilities now:

• He does not write this assumption down in the book. Then it is an error in the book.

• He does write it down in form of a 'global' remark like 'whenever I write $+ \epsilon$ then the $\epsilon$ are iid normally distributed with mean zero unless stated otherwise'. Then IMHO it is bad style because it causes exactly the confusion that you feel right now. That is why I tend to write the assumptions in some shortened form in every Theorem. Only then every building block can be viewed cleanly in its own right.

  • He does write it down closely to the part you are quoting and you/we just did not notice it (also a possibility :-))

However, also in a strict mathematical sense, the normal error is something canonical (the distribution with the highest entropy [once the variance is fixed], hence, producing the strongest models) so that some authors tend to skip this assumption but use in nontheless. Formally, you are absolutely right: They are using mathematics in the "wrong way". Whenever they want to come up with the equation for the density $f_{Y|X}$ as stated above then they need to know $\epsilon$ pretty well, otherwise you just have properties of it flying around in every senseful equation that you try to write down.