# Likelihood vs conditional distribution for Bayesian analysis

We can write Bayes' theorem as

$$p(\theta|x) = \frac{f(X|\theta)p(\theta)}{\int_{\theta} f(X|\theta)p(\theta)d\theta}$$

where $p(\theta|x)$ is the posterior, $f(X|\theta)$ is the conditional distribution, and $p(\theta)$ is the prior.

or

$$p(\theta|x) = \frac{L(\theta|x)p(\theta)}{\int_{\theta} L(\theta|x)p(\theta)d\theta}$$

where $p(\theta|x)$ is the posterior, $L(\theta|x)$ is the likelihood function, and $p(\theta)$ is the prior.

My question is

1. Why is Bayesian analysis done using the likelihood function and not the conditional distribution?
2. Can you say in words what the difference between the likelihood and conditional distribution is? I know the likelihood is not a probability distribution and $L(\theta|x) \propto f(X|\theta)$.
-
There is no difference! The likelihood is the conditional distribution $f(X | \theta)$, well, is proportional to, which is all that matters. –  kjetil b halvorsen Sep 17 '12 at 2:02
Prior parameter $\Theta$ has density $p_\Theta(\theta)$. if the realization of $\Theta$ has value $\theta$ while $x$ is the observed value of a random variable $X$, then the value of the likelihood function $L(\theta\mid x)$ is precisely $f(x\mid \theta)$, the value of the conditional density $f_{X\mid\Theta}(x\mid\Theta=\theta)$ of $X$. The difference is that $$\int_{-\infty}^{\infty}f_{X\mid\Theta}(x\mid\Theta=\theta)dx=1$$ for all realizations of $\Theta$. However, as a function of $\theta$ (and fixed $x$), $L(\theta\mid x)$ is not a density: $$\int L(\theta\mid x)d\theta\neq 1$$ –  Dilip Sarwate Sep 17 '12 at 2:08

Suppose that you have $X_1,\dots,X_n$ random variables (whose values will be observed in your experiment) that are conditionally independent, given that $\Theta=\theta$, with conditional densities $f_{X_i\mid\Theta}(\,\cdot\mid\theta)$, for $i=1,\dots,n$. This is your (postulated) statistical (conditional) model, and the conditional densities express, for each possible value $\theta$ of the (random) parameter $\Theta$, your uncertainty about the values of the $X_i$'s, before you have access to any real data. With the help of the conditional densities you can, for example, compute conditional probabilities like $$P\{X_1\in B_1,\dots,X_n\in B_n\mid \Theta=\theta\} = \int_{B_1\times\dots\times B_n} \prod_{i=1}^n f_{X_i\mid\Theta}(x_i\mid\theta)\,dx_1\dots dx_n \, ,$$ for each $\theta$.

After you have access to an actual sample $(x_1,\dots,x_n)$ of values (realizations) of the $X_i$'s that have been observed in one run of your experiment, the situation changes: there is no longer uncertainty about the observables $X_1,\dots,X_n$. Suppose that the random $\Theta$ assumes values in some parameter space $\Pi$. Now, you define, for those known (fixed) values $(x_1,\dots,x_n)$ a function $$L_{x_1,\dots,x_n} : \Pi \to \mathbb{R} \,$$ by $$L_{x_1,\dots,x_n}(\theta)=\prod_{i=1}^n f_{X_i\mid\Theta}(x_i\mid\theta) \, .$$ Note that $L_{x_1,\dots,x_n}$, known as the "likelihood function" is a function of $\theta$. In this "after you have data" situation, the likelihood $L_{x_1,\dots,x_n}$ contains, for the particular conditional model that we are considering, all the information about the parameter $\Theta$ contained in this particular sample $(x_1,\dots,x_n)$. In fact, it happens that $L_{x_1,\dots,x_n}$ is a sufficient statistic for $\Theta$.

Answering your question, to understand the differences between the concepts of conditional density and likelihood, keep in mind their mathematical definitions (which are clearly different: they are different mathematical objects, with different properties), and also remember that conditional density is a "pre-sample" object/concept, while the likelihood is an "after-sample" one. I hope that all this also help you to answer why Bayesian inference (using your way of putting it, which I don't think is ideal) is done "using the likelihood function and not the conditional distribution": the goal of Bayesian inference is to compute the posterior distribution, and to do so we condition on the observed (known) data.

-
I think Zen's answer really tells you how conceptually the likelihood function and the joint density of values of random variables differ. Still mathematically as a function of both the x$_i$s and θ they are the same and in that sense the likelihood can be looked at as a probability density. The difference you point to in the formula for the Bayes posterior distribution is just a notational difference. But the subtlety of the difference is nicely explained in Zen's answer.