In Linear Regression, when we model $Y$'s dependence on input $X$, we say that $$Y = X\beta +\epsilon $$ where $\epsilon \sim \mathcal{N}(0, \sigma ^2 I)$ where $I$ is the identity matrix. I understand the motivation for why $\epsilon$ should have mean $0$ because we can append a $1$ to every entry of $X$ in order to make $\epsilon$ shift to 0, but why must the variance be $\sigma^2 I$? In particular, why must the covariance between the Gaussian noise of components of $Y$ be 0?
2
-
5$\begingroup$ It doesn't have to be zero--this is called generalized least squares. But if the covariance is nonzero, you need to specify what you think it is or find a way to estimate it, don't you? $\endgroup$– whuber ♦Commented Oct 24, 2017 at 21:01
-
$\begingroup$ It's mainly out of convenience. You could make the variance matrix $\sigma^2 V$, where $V$ is a positive-definite matrix. whuber mentioned GLS - this is exactly what it is. $\endgroup$– ClarinetistCommented Oct 25, 2017 at 14:50
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
You are referring to a very particular setup of the linear regression. It's very strict on requirements to errors. This setup is used in the textbooks because it is easy to work with. Unfortunately, in practice the requirements on errors are difficult to satisfy.
For instance, you imply that in Linear Regression the errors must be Gaussian. That's not necessary. It helps when they are, especially with small sample properties, but non Gaussian errors work fine too. Particularly, to get the coefficient estimates you don't need errors to be Gaussian, but to estimate the variances of coefficients in small samples it helps to have Gaussian errors.
Similarly, with spherical error assumption $Var[\varepsilon|X]=\sigma^2\mathbb I$ where you require that errors are uncorrelated and their variance is constant, it is nice to have but not that necessary. When this assumption holds you get nice properties of the least squares estimator, such as efficiency. Otherwise, you don't need this assumption to get the coefficients with good properties. Also, there are ways to deal with non-spherical errors such as generalized least squares.
