I've been reading Probabilistic Machine Learning, by Kevin Patrick Murphy but I don't quite get the motivation for presenting Machine Learning from a probabilistic point of view. For example, when presenting Linear Regression (or any other model for that matter) it goes like this:

The key property of the model is that the expected value of the output is assumed to be a linear function of the input, $E[y|x] = \vec{w}\cdot\vec{x}$, which makes the model easy to interpret, and easy to fit to data.

The term “linear regression” usually refers to a model of the following form: $p(y|x,\theta)=\mathcal{N}(y|w_0 + \vec{w}\cdot\vec{x},σ^2)$ where $\theta = (w_0, w, σ^2)$ are all the parameters of the model. (In statistics, the parameters $w_0$ and $w$ are usually denoted by $\beta_0$ and $\beta$.)

Where $\mathcal{N}$ is the Gaussian distribution.

Why is he choosing to go with the Gaussian? you could actually use any other distribution and modify the $\vec{w}\cdot\vec{x}$ linear function so that when one computes the expected value of $y|x$ you still get $E[y|x] = \vec{w}\cdot\vec{x}$. For example in the Exponential distribution $f(y, \lambda) = \lambda \exp{(-\lambda y)}$ we can set $\lambda = (\vec{w}\cdot\vec{x})^{-1}$ so that $E[y|x] = \vec{w}\cdot\vec{x}$. And in theory we could do that for other probability distributions (even if not always).

With this change I can see that the Negative Log-likelihood expression would change. However, it still holds true that minimising it would give you the probability distribution that is most similar to the empirical one as per the KL Divergence.

I can only guess that maybe the closest probability distribution that is Exponential like is different from the closest probability distribution that is Gaussian like, but still, the author didn't use this argument at any point to justify the election of the Gaussian in the first place.


1 Answer 1


Sure we can, and we do that with generalized linear models. So why linear regression? It's the simplest possible model that has a closed-form solution to estimate its parameters, that's a huge advantage. Maximizing the Gaussian likelihood is equivalent to minimizing squared error and we have some good threads regarding why squared error is that popular, e.g. Why is the squared difference so commonly used? or What makes mean square error so good?. Also, notice that calculating the mean minimizes the squared error, so it also makes sense for a model that calculates conditional mean (linear regression) to minimize squared error.

  • 1
    $\begingroup$ +1. It seems to me like the first sentence is key: the OP seems to be looking for GLMs. $\endgroup$ May 18 at 13:13
  • $\begingroup$ Thank so much for the answer. So could we say that the motivation for formulating these models in terms of pdf's is to "realize" that we can use different NLL's for the same problem, i.e. I would've never thought of using the NLL expression the Exponential pdf yields if I hadn't formulated the model in terms of pdf's first. Does it make sense? Just trying to understand the motivation behind it. (Because I can think of all these nice properties of the MSE without knowing its relation to the Gaussian likelihood) $\endgroup$
    – CMB
    May 18 at 15:17
  • $\begingroup$ @CMB I don't understand your comment so cannot comment on it. $\endgroup$
    – Tim
    May 18 at 15:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.