As I recalled, we can also assume the data points come from Laplace distribution and hence it will be the linear regression with absolute error.
Why did so many texts assume the data points came from a normal distribution then?
$\begingroup$
$\endgroup$
4
-
3$\begingroup$ The texts that do are to be avoided. What can sometimes be assumed normal is the error term, not "the data". Also, there are a ton of related questions, some of them among these. Yours is likely a duplicate. $\endgroup$– Richard HardyCommented Mar 27, 2022 at 17:05
-
$\begingroup$ @RichardHardy: It's common enough for data to be normally distributed. Error distributions are more of a special case, where the assumption is even stronger: normal, with zero mean. $\endgroup$– MSaltersCommented Mar 28, 2022 at 13:38
-
$\begingroup$ @MSalters, it may or may not be common, but it does not matter for this question. Focusing on normality of the data in linear regression is simply misleading. $\endgroup$– Richard HardyCommented Mar 28, 2022 at 15:35
-
$\begingroup$ This question isn't answerable. A complete answer would provide a collection of all lemmas and theorems for linear regression, and a comparison between all of them. One would need to compare the difficulty in their proofs, as well as a comparison of their subjective utility. Also, I'm going to echo @RichardHardy's concerns about it being a duplicate. $\endgroup$– TaylorCommented Mar 28, 2022 at 18:04
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
3
There is a small point of clarification here: We actually don't need the error term to be Gaussian (see the Gauss-Markov Theorem), but if we assume it is then some nice connections can be made.
Linear regression is commonly motivated by minimizing the squared error loss. Squared error is perhaps the simplest loss function to motivate such a model with, so let's go with it for now. It turns out that minimizing the squared error loss is the same as assuming the likelihood for $y\vert x$ is Gaussian, hence the normality of the error term.
answered Mar 27, 2022 at 12:56
-
$\begingroup$ I think the normality of the error term is not in the same context as this question. The reason for the normality assumption of the error term is mainly by the central limit theorem, as noise is the accumulation of a large number of small little factors. @Demetri Pananos $\endgroup$– NothingCommented Mar 27, 2022 at 15:13
-
1$\begingroup$ But I found the reason for assuming normal data distribution from Gauss-Markov Theorem you mentioned, thanks! $\endgroup$– NothingCommented Mar 27, 2022 at 15:15
-
3$\begingroup$ @Nothing Woolridge notes that justifying the normality assumption via the CLT is not without weakness. First, the factors effecting the error term in may have very different distributions in the population, and the number of factors effecting the error plays effects appropriate the normal approx is. Second, justification using the CLT means that all factors are assumed to effect the error in an additive fashion. If the error is a complicated function of unobserved factors then the CLT does not apply. The connection between minimizing squared error and the gaussian loglik is more tenable. $\endgroup$ Commented Mar 27, 2022 at 17:50
$\begingroup$
$\endgroup$
3
The magic word that nobody seems to have mentioned in this thread is efficiency: we know that if the residuals in a linear regression model $ y = X \beta + \varepsilon $ are normally distributed then the usual OLS estimate of $ \hat \beta = (X^T X)^{-1} X^T y $ is efficient (because it's a maximum likelihood estimator).
The OLS estimator still "works" in the sense of being consistent if $ \varepsilon $ is not normally distributed as long as strict exogeneity is satisfied, i.e. as long as $ \mathbb E[\varepsilon \vert X] = 0 $, since
$$ \mathbb E[\hat \beta] = \mathbb E[(X^T X)^{-1} X^T X \beta + (X^T X)^{-1} X^T \varepsilon] = \beta + \mathbb E[ \mathbb E[(X^T X)^{-1} X^T \varepsilon \vert X]] $$ $$ = \beta + \mathbb E[ (X^T X)^{-1} X^T \mathbb E[ \varepsilon \vert X]] = \beta $$
So all you need to use OLS safely is strict exogeneity, but OLS only makes the most efficient use of the data available if $ \varepsilon $ is normally distributed.
-
1$\begingroup$ It depends on whether you define $\beta$ as a causal parameter or not. A linear model does not require it to be causal, and OLS works just fine for a model that says $\mathbb{E}(Y|X)=\beta X$. Exogeneity is needed only if you define beta as satisfying $\mathbb{E}(Y|\text{do}(X))=\beta X$. $\endgroup$ Commented Mar 28, 2022 at 15:38
-
$\begingroup$ @RichardHardy I'm using "strict exogeneity" in the purely correlational sense here, no causal machinery is invoked. If $ \mathbb E(Y \vert X) = \beta X $ then in my book your setup satisfies strict exogeneity. $\endgroup$ Commented Mar 29, 2022 at 10:30
-
1$\begingroup$ We might be using different definitions of exogeneity. E.g. Chen & Pearl "Regression and causation: a critical examination of six econometrics textbooks" (2013) write in footnote 5: From a causal analytic perspective, X is exogeneous if E[Y |X] = E[Y |do(X) (Pearl, 2000). But I guess there can be other definitions. $\endgroup$ Commented Mar 29, 2022 at 10:41
$\begingroup$
$\endgroup$
The main difference with or without normally distributed errors is whether the inference procedures (standard errors, p-values, t-tests etc) hold in small or large samples.
The inference results hold:
- If errors are (assumed to be) normally distributed, the normal distribution for the estimates is exact, i.e. it holds even in small samples
- Without the normal assumption, using a normal or student distribution for the estimates is an approximation based on an asymptotic result
$\begingroup$
$\endgroup$
Linear regression, like least square regression, does not assume Gaussian distributed error terms.
The assumption of Gaussian distributed error terms relates to statistics like hypothesis tests or computations of confidence intervals. For these types of statistics, it is necessary to have more precise assumptions about the hypothetical distribution of the error. And even for these types of statistics the assumption of Gaussian distribution is not really necessary. What we need is that it is approximately Gaussian distributed (Which statistical analysis should I perform if the data sets are not normally distributed?).
answered Mar 28, 2022 at 10:29