2
$\begingroup$

In an introductory video on Bayesian methods for classification (see reference at the end), the lecturer explains that training involves the computation of the following posterior distribution:

$$ P(\theta|X, y) = \frac{P(y|X,\theta)P(\theta)}{P(y|X)} $$

To make sure we're on the same page, the terms are:

  • $\theta$ are the parameters we're trying to learn
  • $(X, y)$ are the data (X are the feature vectors, y are the labels)
  • $P(\theta)$ is the prior belief in the parameters (e.g. that they're around 0 and so on)

The lecturer claims that this is an application of Bayes' rule, which I'm not able to verify. Could you point me in the right direction? Here's what I'm doing:

$$ \begin{align} P(\theta|X, y) &= \frac{P(\theta, X, y)}{P(X, y)} \text{(conditional prob. definition)}\\ & = \frac{P(\theta, X, y)}{P(y, X)} \text{(conjunction commutes)}\\ & = \frac{P(\theta, X, y)}{P(y|X)P(X)} \text{(chain rule)}\\ &= \frac{P(y|X, \theta)P(X, \theta)}{P(y|X)P(X)} \text{(chain rule)}\\ &= \frac{P(y|X, \theta)P(\theta|X)P(X)}{P(y|X)P(X)} \text{(chain rule)}\\ &= \frac{P(y|X, \theta)P(\theta|X)}{P(y|X)} \text{($P(X)$s cancel)}\\ \end{align} $$

Not only does this not coincide with the lecturer's formula, the term $P(\theta|X)$ seems even more difficult to compute than $P(\theta|X,y)$ because we can't look at the ground truths $y$ in computing $\theta$ anymore. Regardless of whether this last observation is true, I'm primarily interested in a derivation for the lecturer's formula.

Thanks for your time.

References

Bayesian Methods for Machine Learning, Coursera, Week 1, "Bayesian approach to statistics".

$\endgroup$
1
$\begingroup$

\begin{align} P(\theta|X,y)&=\frac{P(\theta, y|X)}{P(y|X)}\\ &=\frac{P(y|X,\theta)P(\theta)}{P(y|X)} \end{align}

$X$ is always in the conditional part in both sides of the expression so there is no reason to find a joint distribution involving $X$.

Your derivation is also correct. The prior information is however independent from $X$(hence called 'prior'). So $P(\theta|X) = P(\theta)$.

$\endgroup$
  • $\begingroup$ Thank you very much. Unable to vote this up due to lack of reputation, but I appreciate the thoughtful response. (I should probably spend some time doing exercises with conditional probabilities: those inferences aren't obvious to me, e.g. that $P(\theta|X, y) = \frac{P(\theta,y|X)}{P(y|X)}$.) $\endgroup$ – Readingtao Jan 28 '18 at 19:23
  • $\begingroup$ This is basically the definition of conditional probability. You don't necessarily have to take all variables and divide their joint distribution by the distribution of all the variables on the "given side" of the probability(as you did in your first step above). It is possible to just apply the rule on a subset of the variables, as long as you leave the rest in the conditioning side. Also, as I said, your derivation is also correct. Please accept the answer if you feel that I have answered your question $\endgroup$ – Andreas G. Jan 28 '18 at 19:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.