In Deep Learning Chapter 5.6, Bayesian Linear Regression is introduced. I'm confused by the following formula:

$$ p(w | X, Y) \propto P(Y | X, w) P(w)$$

  • $X$ is a sample vector input data.
  • $Y$ is the corresponding vector of output data.
  • w is the scalar we want to estimate the linear function.

How is this derived?

My attempts to derive this result in the following:

$$ \begin{align} P(w | X, Y) & = \frac{P(w, X, Y)}{P(X, Y)} \\ & = \frac{P(X)P(w|X)P(Y|w, X)}{P(X)P(Y|X)}\\ & \propto P(w|X)P(Y | X, w) \end{align}$$

  • Apply the definition of conditional probability.
  • Use the chain rule to expand numerator and denominator.
  • Use proportionality to throw away all parts only involving $X$ and $Y$.

My derivation results in $P(w|X)$, where the book reads $P(w)$.

Note: This example features univariate regression, whereas the book actually introduces multivariate regression. The derivation, however, should remain the same. Just change $w$ to a vector and $X$ to a matrix.

  • 1
    $\begingroup$ You are formally correct. In regression, from both Bayesian and classical perspectives, $X$ is conditioned upon, which means that $P(w|X)=P(w)$. For instance, when using a $g$-prior, the variance of $w$ may depend on $X^\text{T}X$. $\endgroup$ – Xi'an Jun 16 '18 at 14:49
  • 1
    $\begingroup$ Aah, that makes sense. And from a Bayesian perspective, $P(w)$ is then the prior, correct? Do want to make your comment an answer so I can accept it? $\endgroup$ – Robert Hönig Jun 16 '18 at 16:53

In a linear regression like $Y=wX+\epsilon$, i.e., when $$\mathbb{E}[Y|X]=wX\quad\text{and}\quad\text{var}(Y|X)=\sigma^2$$the model is defined conditional on $X$, whether or not $X$ is considered as a random variable. There is thus no reason for the prior on the coefficient $w$ not to depend on $X$ and there are many arguments for dependence, one being that the scale of $X$ impacts the scale of $w$ (as, e.g., when moving from meters to kilometres in the recording of the components of $X$). The most classical illustration is Arnold Zellner's so-called $g$-prior that sets $$p(w|X,\tau)\propto\exp\{-w^\text{T}(X^\text{T}X)w/2\tau^2\}\qquad p(\tau)\propto\tau^{-1}$$and relies on the OLS projection matrix $(X^\text{T}X)^{-1}$ as the variance of the Normal prior. In our book Bayesian Essentials with R, this approach is the one recommended for Bayesian linear regression.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.