# The connection between linear regression and Gaussians

I am reading Kevin Murphy's Machine Learning: A Probabilistic Perspective and I am having difficulties understanding Equation 1.5.

Murphy introduces the linear function $$y(\mathbf{x}) = \mathbf{w}^T\mathbf{x} + \epsilon = \sum_{j=1}^D w_j x_j + \epsilon,$$ where $$\mathbf{w}^T\mathbf{x}$$ represents the inner product between the input vector $$\mathbf{x}$$ and the model's weight vector $$\mathbf{w}$$, and $$\epsilon$$ is the residual error. Then Murphy mentions that we often assume that $$\epsilon$$ has a Gaussian distribution with mean $$\mu$$ and variance $$\sigma^2$$.

Murphy makes the connection between linear regression and Gaussians with Equation 1.5: $$p(y|\mathbf{x, \theta}) = \mathcal{N}(y|\mu(\mathbf{x}), \sigma^2(\mathbf{x})).$$

I read this as the probability of getting output $$y$$, given a data point $$\mathbf{x}$$ and some model parameters $$\mathbf{w}$$ and $$\sigma^2$$, is normally distributed with mean $$y$$ given the mean of $$\mathbf{x}$$ (I don't understand this) and variance of $$\mathbf{x}$$.

To me, it seems that $$y$$ should be normally distributed with mean $$\mathbf{w}^T\mathbf{x}$$ and variance $$\sigma^2$$ only if $$\epsilon$$ is normally distributed with mean $$0$$ variance $$\sigma^2$$, i.e., if $$\epsilon \sim \mathcal{N}(0, \sigma^2),$$ then $$p(y|\mathbf{x, \theta}) = \mathcal{N}(\mathbf{w}^T\mathbf{x}, \sigma^2).$$

• I haven't read the book, but I think you are both right, the mu() bit is a bit confusing, it probably means the expectation of y given the modeled x (less the error) Commented May 12, 2021 at 10:11
• $y | \mu(x)$ is the more general idea, as the mean function $\mu(x)$ can be pretty complicated. But in this context the mean is relatively simple, a linear function of the inputs. So, without reading the book I'd guess the author is just going back and forth a little between general ideas and particular examples. Commented May 12, 2021 at 10:18
• Read $\mu(x)$ like $f(x)$ not like $\bar{x}$ because $\mu$ names whatever function provides the conditional expectation of y as a function of x. Commented May 12, 2021 at 10:49
• There are cases where you do not have constant variance $\sigma^{2}$ and you variance is directly affected by your observed $x_{i}$. An example of such case can be found here online.stat.psu.edu/stat462/node/186. So, in order to write the regression in the most general expression it denotes the variance as $\sigma^{2}(x)$. Commented May 12, 2021 at 11:06
• Looks like a mistake (typo) in the book to me. Conditioning on a constant (like $\mu(x)$) has no effect. The author just put the $\mu$ in the wrong place. The mean of the distribution is really $\mu(y|x)$. Commented May 25, 2021 at 23:06

As commented by Nick Cox, I'd suspect that you understand the special case of standard linear regression properly, but the equation 1.5 involving $$\mu(x)$$ and $$\sigma^2(x)$$ regards a more general case, of which the standard linear model is a special case with $$\mu(x)=w^Tx$$ and $$\sigma^2(x)=\sigma^2$$ constant.