1
$\begingroup$

I am reading Kevin Murphy's Machine Learning: A Probabilistic Perspective and I am having difficulties understanding Equation 1.5.

Murphy introduces the linear function $$ y(\mathbf{x}) = \mathbf{w}^T\mathbf{x} + \epsilon = \sum_{j=1}^D w_j x_j + \epsilon, $$ where $\mathbf{w}^T\mathbf{x}$ represents the inner product between the input vector $\mathbf{x}$ and the model's weight vector $\mathbf{w}$, and $\epsilon$ is the residual error. Then Murphy mentions that we often assume that $\epsilon$ has a Gaussian distribution with mean $\mu$ and variance $\sigma^2$.

Murphy makes the connection between linear regression and Gaussians with Equation 1.5: $$ p(y|\mathbf{x, \theta}) = \mathcal{N}(y|\mu(\mathbf{x}), \sigma^2(\mathbf{x})). $$

I read this as the probability of getting output $y$, given a data point $\mathbf{x}$ and some model parameters $\mathbf{w}$ and $\sigma^2$, is normally distributed with mean $y$ given the mean of $\mathbf{x}$ (I don't understand this) and variance of $\mathbf{x}$.

To me, it seems that $y$ should be normally distributed with mean $\mathbf{w}^T\mathbf{x}$ and variance $\sigma^2$ only if $\epsilon$ is normally distributed with mean $0$ variance $\sigma^2$, i.e., if $$ \epsilon \sim \mathcal{N}(0, \sigma^2), $$ then $$ p(y|\mathbf{x, \theta}) = \mathcal{N}(\mathbf{w}^T\mathbf{x}, \sigma^2). $$

$\endgroup$
6
  • $\begingroup$ I haven't read the book, but I think you are both right, the mu() bit is a bit confusing, it probably means the expectation of y given the modeled x (less the error) $\endgroup$ May 12 '21 at 10:11
  • 1
    $\begingroup$ $y | \mu(x)$ is the more general idea, as the mean function $\mu(x)$ can be pretty complicated. But in this context the mean is relatively simple, a linear function of the inputs. So, without reading the book I'd guess the author is just going back and forth a little between general ideas and particular examples. $\endgroup$
    – Nick Cox
    May 12 '21 at 10:18
  • $\begingroup$ Read $\mu(x)$ like $f(x)$ not like $\bar{x}$ because $\mu$ names whatever function provides the conditional expectation of y as a function of x. $\endgroup$ May 12 '21 at 10:49
  • $\begingroup$ There are cases where you do not have constant variance $\sigma^{2}$ and you variance is directly affected by your observed $x_{i}$. An example of such case can be found here online.stat.psu.edu/stat462/node/186. So, in order to write the regression in the most general expression it denotes the variance as $\sigma^{2}(x)$. $\endgroup$
    – Fiodor1234
    May 12 '21 at 11:06
  • $\begingroup$ Looks like a mistake (typo) in the book to me. Conditioning on a constant (like $\mu(x)$) has no effect. The author just put the $\mu$ in the wrong place. The mean of the distribution is really $\mu(y|x)$. $\endgroup$ May 25 '21 at 23:06
1
$\begingroup$

As commented by Nick Cox, I'd suspect that you understand the special case of standard linear regression properly, but the equation 1.5 involving $\mu(x)$ and $\sigma^2(x)$ regards a more general case, of which the standard linear model is a special case with $\mu(x)=w^Tx$ and $ \sigma^2(x)=\sigma^2$ constant.

$\endgroup$
1
$\begingroup$

This is because linear regression can be seen in two different ways, an Ordinary Least Squares approach or a Gaussian distribution approach. The second one is related to Generalized Linear Models and you can find a good explanation in the next video.

https://www.youtube.com/watch?v=UdADuHJUX6Q&t=336s

(had the same question a while ago and this video helped me)

BTW, Kevin has a new book called Probabilistic Machine Learning and you can download it from his github page: https://probml.github.io/pml-book/book1.html

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.