# Conditional probability of posterior distribution for bayesian linear regression

In Deep Learning Chapter 5.6, Bayesian Linear Regression is introduced. I'm confused by the following formula:

$$p(w | X, Y) \propto P(Y | X, w) P(w)$$

• $X$ is a sample vector input data.
• $Y$ is the corresponding vector of output data.
• w is the scalar we want to estimate the linear function.

How is this derived?

My attempts to derive this result in the following:

\begin{align} P(w | X, Y) & = \frac{P(w, X, Y)}{P(X, Y)} \\ & = \frac{P(X)P(w|X)P(Y|w, X)}{P(X)P(Y|X)}\\ & \propto P(w|X)P(Y | X, w) \end{align}

• Apply the definition of conditional probability.
• Use the chain rule to expand numerator and denominator.
• Use proportionality to throw away all parts only involving $X$ and $Y$.

My derivation results in $P(w|X)$, where the book reads $P(w)$.

Note: This example features univariate regression, whereas the book actually introduces multivariate regression. The derivation, however, should remain the same. Just change $w$ to a vector and $X$ to a matrix.

• You are formally correct. In regression, from both Bayesian and classical perspectives, $X$ is conditioned upon, which means that $P(w|X)=P(w)$. For instance, when using a $g$-prior, the variance of $w$ may depend on $X^\text{T}X$. – Xi'an Jun 16 '18 at 14:49
• Aah, that makes sense. And from a Bayesian perspective, $P(w)$ is then the prior, correct? Do want to make your comment an answer so I can accept it? – Robert Hönig Jun 16 '18 at 16:53

In a linear regression like $Y=wX+\epsilon$, i.e., when $$\mathbb{E}[Y|X]=wX\quad\text{and}\quad\text{var}(Y|X)=\sigma^2$$the model is defined conditional on $X$, whether or not $X$ is considered as a random variable. There is thus no reason for the prior on the coefficient $w$ not to depend on $X$ and there are many arguments for dependence, one being that the scale of $X$ impacts the scale of $w$ (as, e.g., when moving from meters to kilometres in the recording of the components of $X$). The most classical illustration is Arnold Zellner's so-called $g$-prior that sets $$p(w|X,\tau)\propto\exp\{-w^\text{T}(X^\text{T}X)w/2\tau^2\}\qquad p(\tau)\propto\tau^{-1}$$and relies on the OLS projection matrix $(X^\text{T}X)^{-1}$ as the variance of the Normal prior. In our book Bayesian Essentials with R, this approach is the one recommended for Bayesian linear regression.