Rationale behind choosing Gaussian probability distribution for Linear Regression

Question

I've been reading Probabilistic Machine Learning, by Kevin Patrick Murphy but I don't quite get the motivation for presenting Machine Learning from a probabilistic point of view. For example, when presenting Linear Regression (or any other model for that matter) it goes like this:

The key property of the model is that the expected value of the output is assumed to be a linear function of the input, $E[y|x] = \vec{w}\cdot\vec{x}$, which makes the model easy to interpret, and easy to fit to data.

The term “linear regression” usually refers to a model of the following form: $p(y|x,\theta)=\mathcal{N}(y|w_0 + \vec{w}\cdot\vec{x},σ^2)$ where $\theta = (w_0, w, σ^2)$ are all the parameters of the model. (In statistics, the parameters $w_0$ and $w$ are usually denoted by $\beta_0$ and $\beta$.)

Where $\mathcal{N}$ is the Gaussian distribution.

Why is he choosing to go with the Gaussian? you could actually use any other distribution and modify the $\vec{w}\cdot\vec{x}$ linear function so that when one computes the expected value of $y|x$ you still get $E[y|x] = \vec{w}\cdot\vec{x}$. For example in the Exponential distribution $f(y, \lambda) = \lambda \exp{(-\lambda y)}$ we can set $\lambda = (\vec{w}\cdot\vec{x})^{-1}$ so that $E[y|x] = \vec{w}\cdot\vec{x}$. And in theory we could do that for other probability distributions (even if not always).

With this change I can see that the Negative Log-likelihood expression would change. However, it still holds true that minimising it would give you the probability distribution that is most similar to the empirical one as per the KL Divergence.

I can only guess that maybe the closest probability distribution that is Exponential like is different from the closest probability distribution that is Gaussian like, but still, the author didn't use this argument at any point to justify the election of the Gaussian in the first place.

Answer 1

Sure we can, and we do that with generalized linear models. So why linear regression? It's the simplest possible model that has a closed-form solution to estimate its parameters, that's a huge advantage. Maximizing the Gaussian likelihood is equivalent to minimizing squared error and we have some good threads regarding why squared error is that popular, e.g. Why is the squared difference so commonly used? or What makes mean square error so good?. Also, notice that calculating the mean minimizes the squared error, so it also makes sense for a model that calculates conditional mean (linear regression) to minimize squared error.