Skip to main content
6 of 6
added 29 characters in body
Aksakal
  • 62.3k
  • 6
  • 106
  • 206

The Gaussian assumption, if any, is on the errors of the process, not the regressors. In the simplest case the model is $$y_i=\alpha+\beta x_i+\varepsilon_i$$, where $\varepsilon_i\sim\mathcal N(0,\sigma^2)$, i.e. errors are Gaussian.

If you knew true parameters of the model you could infer the errors $\varepsilon_i=y_i-(\alpha+\beta x_i)$. You could interpret this as measured value $y_i$ being a random variable with mean $\alpha+\beta x_i$, i.e. $y_i\sim\mathcal N(\alpha+\beta x_i,\sigma^2)$. Notice how we replace a constant $\mu$ in a textbook Gaussian distribution expression with the model value $(\alpha+\beta x_i)$.

The errors are not observed, so we can estimate them as residuals $\hat \varepsilon_i\equiv r_i=y_i-(a+b x_i)$. Then we write the MLE equation and estimate $a,b,\hat\sigma$ from it. The estimates are of true parameters $\alpha,\beta,\sigma$ that you denoted collectively as $\theta$.

Actually, in many cases we don’t need the errors to be Gaussian IID, because it is enough that they have finite variance and are uncorrelated, plus some other assumptions. We then show that for large enough sample due to CLT the parameter estimates are Gaussian around true parameters. This way the requirements on the dataset are less restrictive and more realistic. Gaussian iid requirement on errors is often unrealistic, and it's a very strong assumption that is hard to test.

Requiring Gaussian assumption on regressors would render the model almost useless in most applications. Imagine you are conducting an experiment and setting X at a few chosen values, e.g. a temperature of the oven at $x=100,200,300,400F$. First, X is not random at all. Second, it’s distribution is not Gaussian. However, the regression can be applied in these type of settings.

Aksakal
  • 62.3k
  • 6
  • 106
  • 206