1
$\begingroup$

I am reading Kevin Murphy's Machine Learning: A Probabilistic Perspective and I am having difficulties understanding Equation 1.5.

Murphy introduces the linear function $$ y(\mathbf{x}) = \mathbf{w}^T\mathbf{x} + \epsilon = \sum_{j=1}^D w_j x_j + \epsilon, $$ where $\mathbf{w}^T\mathbf{x}$ represents the inner product between the input vector $\mathbf{x}$ and the model's weight vector $\mathbf{w}$, and $\epsilon$ is the residual error. Then Murphy mentions that we often assume that $\epsilon$ has a Gaussian distribution with mean $\mu$ and variance $\sigma^2$.

Murphy makes the connection between linear regression and Gaussians with Equation 1.5: $$ p(y|\mathbf{x, \theta}) = \mathcal{N}(y|\mu(\mathbf{x}), \sigma^2(\mathbf{x})). $$

I read this as the probability of getting output $y$, given a data point $\mathbf{x}$ and some model parameters $\mathbf{w}$ and $\sigma^2$, is normally distributed with mean $y$ given the mean of $\mathbf{x}$ (I don't understand this) and variance of $\mathbf{x}$.

To me, it seems that $y$ should be normally distributed with mean $\mathbf{w}^T\mathbf{x}$ and variance $\sigma^2$ only if $\epsilon$ is normally distributed with mean $0$ variance $\sigma^2$, i.e., if $$ \epsilon \sim \mathcal{N}(0, \sigma^2), $$ then $$ p(y|\mathbf{x, \theta}) = \mathcal{N}(\mathbf{w}^T\mathbf{x}, \sigma^2). $$

Improve this question
$\endgroup$
6
  • $\begingroup$ I haven't read the book, but I think you are both right, the mu() bit is a bit confusing, it probably means the expectation of y given the modeled x (less the error) $\endgroup$
    – Dirk N
    May 12, 2021 at 10:11
  • 1
    $\begingroup$ $y | \mu(x)$ is the more general idea, as the mean function $\mu(x)$ can be pretty complicated. But in this context the mean is relatively simple, a linear function of the inputs. So, without reading the book I'd guess the author is just going back and forth a little between general ideas and particular examples. $\endgroup$
    – Nick Cox
    May 12, 2021 at 10:18
  • $\begingroup$ Read $\mu(x)$ like $f(x)$ not like $\bar{x}$ because $\mu$ names whatever function provides the conditional expectation of y as a function of x. $\endgroup$
    – conjugateprior
    May 12, 2021 at 10:49
  • $\begingroup$ There are cases where you do not have constant variance $\sigma^{2}$ and you variance is directly affected by your observed $x_{i}$. An example of such case can be found here online.stat.psu.edu/stat462/node/186. So, in order to write the regression in the most general expression it denotes the variance as $\sigma^{2}(x)$. $\endgroup$
    – Fiodor1234
    May 12, 2021 at 11:06
  • $\begingroup$ Looks like a mistake (typo) in the book to me. Conditioning on a constant (like $\mu(x)$) has no effect. The author just put the $\mu$ in the wrong place. The mean of the distribution is really $\mu(y|x)$. $\endgroup$
    – BigBendRegion
    May 25, 2021 at 23:06

2 Answers 2

Reset to default
1
$\begingroup$

As commented by Nick Cox, I'd suspect that you understand the special case of standard linear regression properly, but the equation 1.5 involving $\mu(x)$ and $\sigma^2(x)$ regards a more general case, of which the standard linear model is a special case with $\mu(x)=w^Tx$ and $ \sigma^2(x)=\sigma^2$ constant.

Improve this answer
$\endgroup$
1
$\begingroup$

This is because linear regression can be seen in two different ways, an Ordinary Least Squares approach or a Gaussian distribution approach. The second one is related to Generalized Linear Models and you can find a good explanation in the next video.

https://www.youtube.com/watch?v=UdADuHJUX6Q&t=336s

(had the same question a while ago and this video helped me)

BTW, Kevin has a new book called Probabilistic Machine Learning and you can download it from his github page: https://probml.github.io/pml-book/book1.html

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.