Conditional probability of posterior distribution for bayesian linear regression

Question

In Deep Learning Chapter 5.6, Bayesian Linear Regression is introduced. I'm confused by the following formula:

$$ p(w | X, Y) \propto P(Y | X, w) P(w)$$

• $X$ is a sample vector input data.
• $Y$ is the corresponding vector of output data.
• w is the scalar we want to estimate the linear function.

How is this derived?

My attempts to derive this result in the following:

$$ \begin{align} P(w | X, Y) & = \frac{P(w, X, Y)}{P(X, Y)} \\ & = \frac{P(X)P(w|X)P(Y|w, X)}{P(X)P(Y|X)}\\ & \propto P(w|X)P(Y | X, w) \end{align}$$

• Apply the definition of conditional probability.
• Use the chain rule to expand numerator and denominator.
• Use proportionality to throw away all parts only involving $X$ and $Y$.

My derivation results in $P(w|X)$, where the book reads $P(w)$.

Note: This example features univariate regression, whereas the book actually introduces multivariate regression. The derivation, however, should remain the same. Just change $w$ to a vector and $X$ to a matrix.

Answer 1

In a linear regression like $Y=wX+\epsilon$, i.e., when $$\mathbb{E}[Y|X]=wX\quad\text{and}\quad\text{var}(Y|X)=\sigma^2$$the model is defined conditional on $X$, whether or not $X$ is considered as a random variable. There is thus no reason for the prior on the coefficient $w$ not to depend on $X$ and there are many arguments for dependence, one being that the scale of $X$ impacts the scale of $w$ (as, e.g., when moving from meters to kilometres in the recording of the components of $X$). The most classical illustration is Arnold Zellner's so-called $g$-prior that sets $$p(w|X,\tau)\propto\exp\{-w^\text{T}(X^\text{T}X)w/2\tau^2\}\qquad p(\tau)\propto\tau^{-1}$$and relies on the OLS projection matrix $(X^\text{T}X)^{-1}$ as the variance of the Normal prior. In our book Bayesian Essentials with R, this approach is the one recommended for Bayesian linear regression.