# Posterior formula for classifier training

In an introductory video on Bayesian methods for classification (see reference at the end), the lecturer explains that training involves the computation of the following posterior distribution:

$$P(\theta|X, y) = \frac{P(y|X,\theta)P(\theta)}{P(y|X)}$$

To make sure we're on the same page, the terms are:

• $\theta$ are the parameters we're trying to learn
• $(X, y)$ are the data (X are the feature vectors, y are the labels)
• $P(\theta)$ is the prior belief in the parameters (e.g. that they're around 0 and so on)

The lecturer claims that this is an application of Bayes' rule, which I'm not able to verify. Could you point me in the right direction? Here's what I'm doing:

\begin{align} P(\theta|X, y) &= \frac{P(\theta, X, y)}{P(X, y)} \text{(conditional prob. definition)}\\ & = \frac{P(\theta, X, y)}{P(y, X)} \text{(conjunction commutes)}\\ & = \frac{P(\theta, X, y)}{P(y|X)P(X)} \text{(chain rule)}\\ &= \frac{P(y|X, \theta)P(X, \theta)}{P(y|X)P(X)} \text{(chain rule)}\\ &= \frac{P(y|X, \theta)P(\theta|X)P(X)}{P(y|X)P(X)} \text{(chain rule)}\\ &= \frac{P(y|X, \theta)P(\theta|X)}{P(y|X)} \text{(P(X)s cancel)}\\ \end{align}

Not only does this not coincide with the lecturer's formula, the term $P(\theta|X)$ seems even more difficult to compute than $P(\theta|X,y)$ because we can't look at the ground truths $y$ in computing $\theta$ anymore. Regardless of whether this last observation is true, I'm primarily interested in a derivation for the lecturer's formula.

References

Bayesian Methods for Machine Learning, Coursera, Week 1, "Bayesian approach to statistics".

$X$ is always in the conditional part in both sides of the expression so there is no reason to find a joint distribution involving $X$.
Your derivation is also correct. The prior information is however independent from $X$(hence called 'prior'). So $P(\theta|X) = P(\theta)$.
• Thank you very much. Unable to vote this up due to lack of reputation, but I appreciate the thoughtful response. (I should probably spend some time doing exercises with conditional probabilities: those inferences aren't obvious to me, e.g. that $P(\theta|X, y) = \frac{P(\theta,y|X)}{P(y|X)}$.) – Readingtao Jan 28 '18 at 19:23