Conjugate prior, unclear definition

Question

Consider the following definition:

A family $\cal F$ of probability distributions on $\Theta$ is said to be conjugate (or closed under sampling) for a likelihood function $f(x|\theta)$ if for every $\pi\in\cal F$, the posterior distribution $\pi(\theta|x)$ also belongs to $\cal F$.

This was a definition. My question is, where we have used again this expression: $$f(x|\theta)$$ in the second part of the definition? We have just exhibited $f(x|\theta)$ and the forgot about it!

Moreover I do not follow is what is actually the conjugate of $\cal F$ that we have just defined? There is a table of natural conjugates of some common exponential families in my source but I do not follow how $\pi(\theta |x)$ and $f(\theta |x)$ are related.

Answer 1

The key point you are missing is that the likelihood is used in the definition of the posterior: $$\pi(\theta|x)=\frac{\pi(\theta)f(x|\theta)}{\int\pi(u)f(x|u)\,du}.$$

This is also often written as $$\pi(\theta|x)\propto\pi(\theta) f(x|\theta)$$ where the $\propto$ symbol means "proportional to".

Answer 2

$\cal F$ is a family of distributions that, upon multiplying with a likelihood of a function class that is conjugate to it, results in an unnormalized posterior distribution that is of the same distribution class as the prior.

---

Let's look at an example: For Bernoulli likelihoods, the prior is the Beta distribution.
https://en.wikipedia.org/wiki/Bernoulli_distribution https://en.wikipedia.org/wiki/Beta_distribution

• likelihood: $Bernoulli(y; p) = p^y (1-p)^{1-y}$, where

Observation $y=0$ or $y=1$ [tails (T) or heads (H)], $p \in [0, 1]$ (probability of heads)

• prior: $Beta(p; a, B) = p^{a-1} (1-p)^{B-1}$, where

its parameters are $a > 0, B > 0$.

• Multiplying the likelihood and prior, you get an unnormalized posterior density

$\mbox{posterior (Beta class)} \propto \mbox{lklhd (Bernoulli class)}\times\mbox{conjg prior (Beta class)}$

$\mbox{posterior} \propto p^{y+a-1} (1-p)^{B-y}$, which is $Beta(p; a+y, B-y+1)$.

================

So let's put some numbers to this. We're going to look at an example where we update our belief about the probability that a coin will show HEADS, starting from before our first flip, and update our belief every time we flip a coin, and see a new observation.

Let's say you start off agnostic on probability of seeing H. The prior is $Beta(1, 1)$. Now you flip the coin 3 times, and observe the sequence {H, H, T}.

Starting with a prior $Beta(1, 1)$, you flip and observe an H. Combining your prior with the likelihood, your posterior is now $Beta(2, 1)$.

Now your posterior becomes your prior, $Beta(2, 1)$, you flip a second time, and observe an H, your posterior is now $Beta(3, 1)$.

Now your posterior becomes your prior, $Beta(3, 1)$, you flip a second time, and observe an T, your posterior is now $Beta(3, 2)$.

The idea here is that you have some prior belief about what the probability of getting H (heads) is, prior to ever flipping a coin. It may be that you are neutral, $a=1$, $B=1$, that is all $p$ equally likely (you have no idea). When you flip the coin once, this gives you some new information. Let's say you flip and get an H. Now this boosts the probability of higher values of $p$.

Note that as soon as you see your first Tail after the 3rd flip, the prior probability of p is now 0 at p=1 - ie there is SOME probability of seeing Tail.

Now note that its perfectly reasonable to have a prior that's say 2 delta functions at $p=0.23$ and $p=0.88$. Combining this prior with a likelihood coming from an observation of an H or T results in some strange function class, which is valid as a posterior as well. As you can see from the above example, using conjugate priors has nice properties, where it might be easy to say build a sequential estimation algorithm that updates the belief about the probable value of $p$ every time you get a new observation. This wouldn't be computationally very easy had you started off with a prior that was 2 delta functions.

Answer 3

Conjugate priors are defined for and only for exponential family likelihoods. That is, if the likelihood writes as $$f(x|\theta) = h(x)\times\exp\{\theta\cdot S(x)-\Psi(\theta)\}$$ against a specific dominating measure in $\mathcal{X}$ and if the prior write as $$\pi(\theta)=\exp\{\theta.S_0-\lambda\Psi(\theta)\}$$ against a specific dominating measure in $\Theta$, then the posterior will satisfy \begin{align*} \pi(\theta|x_1,\ldots,x_n) &\propto \exp\{\theta.S_0-\lambda\Psi(\theta)\}\times \exp\left\{\theta\cdot\sum_{i=1}^nS(x_i)-\Psi(\theta)\right\}\\ &\propto \exp\left\{\theta\cdot\left[S_0+\sum_{i=1}^nS(x_i)\right]-(\lambda+1)\Psi(\theta)\right\} \end{align*} which is indeed of the same shape [against the same specific dominating measure in $\Theta$]. Outside exponential families, the only families of likelihoods that enjoy conjugate priors are exponential families multiplied by a parameter dependent support like Uniform distributions of all kinds. Apart from these the Pitman-Koopman-Darmois lemma excludes the existence of a finite dimension conjugate family of priors.