10
$\begingroup$

I am revising linear regression.

The textbook by Greene states:

enter image description here enter image description here

Now, of course there will be other assumptions on the linear regression model, such as $E(\epsilon|X)=0$. This assumption combined with the linearity assumption (which in effect defines $\epsilon$), puts structure on the model.

However, the linearity assumption by itself does not put any structure on our model, since $\epsilon$ can be completely arbitrary. For any variables $X, y$ whatsoever, no matter what the relation between the two we could define an $\epsilon$ such that the linearity assumption holds. Therefore, the linearity "assumption" should really be called a definition of $\epsilon$, rather than an assumption.

Therefore I am wondering:

  1. Is Greene being sloppy? Should he actually have written: $E(y|X)=X\beta$? This is a "linearity assumption" that actually puts structure on the model.

  2. Or do I have to accept that the linearity assumption does not put structure on the model but only defines an $\epsilon$, where the other assumptions will use that definition of $\epsilon$ to put structure on the model?


Edit: since there seems to be some confusion around the other assumptions, let me add the full set of assumptions here:

enter image description here

This is from Greene, Econometric Analysis, 7th ed. p. 16.

$\endgroup$
  • 4
    $\begingroup$ These are perceptive observations (+1). In all fairness, though, I believe most (if not all) authors are working within a framework in which the very meaning of an additive error like $\epsilon$ includes the assumption that its distribution is centered at $0$. $\endgroup$ – whuber Jan 2 '18 at 13:24
  • 2
    $\begingroup$ @whuber, I've added the whole set of assumptions. look at A3. A3 makes explicit that it is centered at 0, which would imply that Greene does not assume this in A1, which leaves me to question whether A1 has any logical content at all, apart from defining $\epsilon$. $\endgroup$ – user56834 Jan 2 '18 at 17:07
  • 2
    $\begingroup$ The intended meaning of a list of assumptions is that they hold collectively, not separately. This doesn't exhibit any "sloppiness." $\endgroup$ – whuber Jan 2 '18 at 17:18
  • 2
    $\begingroup$ @AdamO, the word "correct" seems not to have a precise meaning to me. I am trying to more exactly understand this. It seems to me that the moest precise formulation of all this is to say that assumption 1 should be called "definition of $\epsilon$", and then everything makes sense. Or I am actually missing something, which is why I asked this question. Unfortunately so far I have not seen a direct answer to that question $\endgroup$ – user56834 Jan 2 '18 at 19:33
  • 2
    $\begingroup$ @Programmer2134 you are getting imprecise answers because you are asking an imprecise question. One does not "put structure on a model" as you say. If the wrong mean model ($f(x)$) is used, then the response is characterized as $Y = f(x) + \text{bias} + \text{error}$. and the residuals are taken as the sum of the bias and the error. $\endgroup$ – AdamO Jan 2 '18 at 19:40
8
$\begingroup$
  1. Is Greene being sloppy? Should he actually have written: $E(y|X)=X\beta$? This is a "linearity assumption" that actually puts structure on the model.

In a sense, yes and no. In the one hand, yes, given current modern causality research he is sloppy, but just like most econometrics textbooks are, in the sense that they do not make a clear distinction of causal and observational quantities, leading to common confusions like this very question. But, in the other hand, no, this assumption is not sloppy in the sense that it is indeed different from simply assuming $E(y|X)=X\beta$.

The crux of the matter here is the difference between the conditional expectation, $E(y|X)$, and the structural (causal) equation of $y$, as well as its structural (causal) expectation $E[Y|do(X)]$. The linearity assumption in Greene is a structural assumption. Let's see a simple example. Imagine the structural equation is:

$$ y= \beta x + \gamma x^2 + \epsilon $$

Now let $E[\epsilon |x] = \delta x - \gamma x^2$. Then we would have:

$$ E[y|x] = \beta'x $$

where $\beta' = \beta + \delta$. Moreover, we can write $y = \beta'x + \epsilon'$ and we would have $E[\epsilon'|x] = 0$. This shows we can have a correctly specified linear conditional expectation $E[y|x]$ which by definition is going to have an orthogonal disturbance, yet the structural equation would be nonlinear.

  1. Or do I have to accept that the linearity assumption does not put structure on the model but only defines an $\epsilon$, where the other assumptions will use that definition of $\epsilon$ to put structure on the model?

The linearity assumption does define an $\epsilon$, that is, $\epsilon := y - X\beta = y - E[Y|do(X)]$ by definition, where $\epsilon$ represents the deviations of $y$ from its expectation when we experimentally set $X$ (see Pearl section 5.4). The other assumptions are used either for identification of the structural parameters (for instance, the assumption of exogeneity of $\epsilon$ allows you to identify the structural expectation $E[Y|do(X)]$ with the conditional expectation $E[Y|X]$) or for derivation of statistical properties of the estimators (for instance, the assumption of homoskedasticity guarantees OLS is BLUE, the assumption of normality makes it easy to derive "finite sample" results for inference etc).

However, the linearity assumption by itself does not put any structure on our model, since $\epsilon$ can be completely arbitrary. For any variables $X, y$ whatsoever, no matter what the relation between the two we could define an $\epsilon$ such that the linearity assumption holds.

Your statement here goes into the main problem of causal inference in general! As shown in the simple example above, we can cook up structural disturbances that could make the conditional expectation of $y$ given $x$ linear. In general, several different structural (causal) models can have the same observational distribution, you can even have causation without observed association. Therefore, in this sense, you are correct --- we need more assumptions on $\epsilon$ in order to put "more structure" into the problem and identify the structural parameters $\beta$ with observational data.

Side note

It's worth mentioning most econometrics textbooks are confusing when it comes to the distinction between regression and structural equations and their meaning. This has been documented lately. You can check a paper by Chen and Pearl here as well as an extended survey by Chris Auld. Greene is one of the books examined.

$\endgroup$
  • $\begingroup$ Thanks, this is the answer I was looking for. So when you say the linearity assumption is a structural assumption, then what does that entail exactly about the causal relationship between $\epsilon$ and $x$? There can still be a causal relationship correct? Its just that the direct causal relationship from $x$ to $y$ is linear, is that it? There can still be a highly nonlinear causal effect of $x$ on $y$ through $\epsilon$? $\endgroup$ – user56834 Jan 3 '18 at 4:54
  • 1
    $\begingroup$ @Programmer2134 that's another area where econometrics textbooks are sloppy, you will find little reference to direct/indirect effects, mediation etc. If the equation is structural, then we can have an operational definition of the structural disturbance as the difference of $y$ with the expected causal effect of $X$, that is $\epsilon := y - E[Y|do(X)] = y - X\beta$. Hence, in this sense, the structural $\epsilon$ is not "caused" by $X$. However, this tells us nothing about the association of $\epsilon$ and $X$, for they could have common causes. $\endgroup$ – Carlos Cinelli Jan 3 '18 at 10:15
  • $\begingroup$ @Programmer2134 by the way, your concerns are on the right track, I think Pearl's Primer on causal inference might be an interesting companion to Greene's! $\endgroup$ – Carlos Cinelli Jan 3 '18 at 10:18
  • $\begingroup$ Incidentally, I started reading "Causality: Models, Reasoning and Inference" by Pearl a while ago. I thought it was very interesting, but it was somewhat abstract for me. I did not get beyond chapter 2. Do you think "primer on causal inference" will be better suited? (i.e. introduce concepts more intuitively). $\endgroup$ – user56834 Jan 3 '18 at 12:02
  • 1
    $\begingroup$ @ColorStatistics you can use regression for forecasting, sure, but then the exogeneity assumption plays no role whatsoever. That’s what the OP started to suspect by himself, by questioning why Greene didn’t simply wrote the assumption as $E(Y|x)$ being linear. $\endgroup$ – Carlos Cinelli Dec 12 '18 at 17:01
0
$\begingroup$

edited after comments by OP and Matthew Drury

To answer this question I assume Greene, and OP, have the following definition of linearity in mind: Linearity means that for every one unit increase in this predictor, the outcome is increased by beta ($β$), wherever on the range of possible predictor values this one-unit increase occurs. I.e. the function $y=f(x)$ is $y=a+bx$ and not e.g. $y=a+bx^2$ or $y=a+sin(x)$. Further, this assumption is focused on the betas and thus applies to predictors (aka independent variables).

The expectation of residuals conditional on the model $E(ϵ|X)$ is something else. Yes, it is true that the math behind a linear regression defines/tries to define $E(ϵ|X)=0$. However, this is usually set over the entire range of fitted/predicted values for $y$. If you look at specific parts of the linear predictor and the predicted value of $y$, you might notice heteroscedasticity (areas where the variation of $ϵ$ is larger than elsewhere), or areas where $E(ϵ|X)≠0$. A non-linear association between the $x$'s and $y$ might be the cause for this, but is not the only reason heteroscedasticity or $E(ϵ|X)≠0$ might occur (see for example missing predictor bias).

From the comments: OP states "the linearity assumption does not restrict the model in any way, given that epsilon is arbitrary and can be any function of XX whatsoever", to which I would agree. I think this is made clear by linear regressions being able to fit to any data, whether or not the linearity assumption is violated or not. I'm speculating here, but that might be the reason why Greene chose to keep the error $ϵ$ in the formula - saving the $E(ϵ|X)=0$ for later - to denote that in assuming linearity, $y$ (and not the expected $y$) can be defined based on $X$ but maintains some error $ϵ$, regardless of what values $ϵ$ takes. I can only hope that he would later go on to state the relevancy of $E(ϵ|X)=0$.

In short (admittedly, without fully reading Greene's book and checking his argumentation):

  1. Greene probably refers to the betas being constant for the entire range of the predictor (emphasis should be placed on the beta in the $y=Xβ + ϵ$ or $E(ϵ|X)=Xβ$ equations;
  2. The linearity assumption does put some structure on the model. You should however note that transformations or additions such as splines prior to modelling, can make non-linear associations conform to the linear regression framework.
$\endgroup$
  • 3
    $\begingroup$ This is helpful, but the appeal to continuity is not needed in any sense. The machinery works the same way if $X$ is just based on $(0, 1)$ predictors. $\endgroup$ – Nick Cox Jan 2 '18 at 14:19
  • 1
    $\begingroup$ You wrote $f(y)$ but I think you meant $f(x)$,. $\endgroup$ – Nick Cox Jan 2 '18 at 14:22
  • $\begingroup$ @NickCox I've edited these points. $\endgroup$ – IWS Jan 2 '18 at 14:23
  • 1
    $\begingroup$ Ehat do you mean by normalcy? If you mean normality then it is incorrect because epsilon does not have to be normal for it to have a conditional expectation of zero. But you mean something else? Also, yes beta is assumed constant for all observations. And what do you think is wrong with my argument that the linearity assumption does not restrict the model in any way, given that epsilon is arbitrary and can be any function of $X$ whatsoever? Note that I know what heteroskedasticity is and that linearity means linear in parameters, not in variables. $\endgroup$ – user56834 Jan 2 '18 at 15:01
  • 3
    $\begingroup$ I disagree with this. The expectation assumption is unrelated to normalcy, but is absolutely needed to make any sense of the structural linearity assumption. Otherwise, as noted by the op, the linearity assumption is meaningless. A normality assumption is a quite different beast, and is often unneeded. $\endgroup$ – Matthew Drury Jan 2 '18 at 15:49
-1
$\begingroup$

I was a little confused by the answer above, hence I'll give it another shot. I think the question is not actually about 'classical' linear regression but about the style of that particular source. On the classical regression part:

However, the linearity assumption by itself does not put any structure on our model

That is absolutely correct. As you have stated, $\epsilon$ might as well kill the linear relation and add up something completely independent from $X$ so that we cannot compute any model at all.

Is Greene being sloppy? Should he actually have written: $E(y|X)=Xβ$

I do not want to answer the first question but let me sum up the assumptions you need for usual linear regression:

Let us assume that you observe (you are given) data points $x_i \in \mathbb{R}^d$ and $y_i \in \mathbb{R}$ for $i=1,...,n$. You need to assume that the data $(x_i, y_i)$ you have observed comes from independently, identically distributed random variables $(X_i, Y_i)$ such that ...

  1. There exists a fixed (independent of $i$) $\beta \in \mathbb{R}^d$ such that $Y_i = \beta X_i + \epsilon_i$ for all $i$ and the random variables $\epsilon_i$ are such that

  2. The $\epsilon_i$ are iid as well and $\epsilon_i$ is distributed as $\mathcal{N}(0, \sigma)$ ($\sigma$ must be independent of $i$ as well)

  3. For $X = (X_1, ..., X_n)$ and $Y = (Y_1, ..., Y_n)$ the variables $X, Y$ have a common density, i.e. the single random variable $(X, Y)$ has a density $f_{X,Y}$

Now you can run down the usual path and compute

$$f_{Y|X}(y|x) = f_{Y,X}(y,x)/f_X(x) = \left(\frac{1}{\sqrt{2\pi d}}\right)^n \exp{\left( \frac{-\sum_{i=1}^n (y_i - \beta x_i)^2}{2\sigma}\right)} $$

so that by the usual 'duality' between machine learning (minimalization of error functions) and probability theory (maximization of likelihoods) you maximize $-\log f_{Y|X}(y|x)$ in $\beta$ which in fact, gives you the usual "RMSE" stuff.

Now as stated: If the author of the book you are quoting wants to make this point (which you have to do if you ever want to be able to compute the 'best possible' regression line in the basic setup) then yes, he must make this assumption on the normalicity of the $\epsilon$ somewhere in the book.

There are different possibilities now:

  • He does not write this assumption down in the book. Then it is an error in the book.

  • He does write it down in form of a 'global' remark like 'whenever I write $+ \epsilon$ then the $\epsilon$ are iid normally distributed with mean zero unless stated otherwise'. Then IMHO it is bad style because it causes exactly the confusion that you feel right now. That is why I tend to write the assumptions in some shortened form in every Theorem. Only then every building block can be viewed cleanly in its own right.

    • He does write it down closely to the part you are quoting and you/we just did not notice it (also a possibility :-))

However, also in a strict mathematical sense, the normal error is something canonical (the distribution with the highest entropy [once the variance is fixed], hence, producing the strongest models) so that some authors tend to skip this assumption but use in nontheless. Formally, you are absolutely right: They are using mathematics in the "wrong way". Whenever they want to come up with the equation for the density $f_{Y|X}$ as stated above then they need to know $\epsilon$ pretty well, otherwise you just have properties of it flying around in every senseful equation that you try to write down.

$\endgroup$
  • 3
    $\begingroup$ the errors do not need to be normally distributed in order to use OLS. $\endgroup$ – user56834 Jan 2 '18 at 17:05
  • $\begingroup$ (-1) The errors do not need to be normally distributed. They in fact do not even need to be independent or identically distributed for the parameter estimate to be unbiased and for the tests to be consistent. Your much more stringent specifications are necessary for OLS to be an exact test. $\endgroup$ – AdamO Jan 2 '18 at 18:21
  • $\begingroup$ @AdamO: Ah? So how do you compute the likelihood then? Or rather... if you are asked to implement linear regression: what regression line do you select if the error is not normally distributed and the single $\epsilon_i$ are not independent? $\endgroup$ – Fabian Werner Jan 2 '18 at 21:44
  • 1
    $\begingroup$ @FabianWerner my choice of model depends on the question which is to be asked. Linear regression estimates a first order trend in a set of data, a "rule of thumb" relating a difference in X to a difference in Y. If the errors are not normally distributed, the Lindeberg Feller CLT guarantees that CIs and PIs are approximately correct in even very small samples. If the errors are non-independent (and the dependence structure is unknown), the estimates are not biased although the SEs may be incorrect. Sandwich error estimation alleviates this problem. $\endgroup$ – AdamO Jan 2 '18 at 22:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.