0
$\begingroup$

I understand that Likelihood is not a probability distribution but what about Marginal likelihood ? Suppose we are given a set of data points $\pmb{X}$ such that each data point $x_{i} \sim p(x_{i}|\theta)$. Now $\theta$ is also a random variable such that $\theta \sim p(\theta|\alpha)$. Then Wikipedia describes marginal likelihood as the probability $p(\pmb{X}|\alpha)$ where $\alpha$ is integrated out(marginalized). Does this mean that all marginal likelihoods integrate to 1 and therefore formally are probability distributions as well ? Or is it just an error in writing the marginal likelihood as $p(\pmb{X}|\alpha)$ for a lack of a better alternative. Also various texts refer to the likelihood as $p(\pmb{X}|\theta)$ , are all of them being non-rigorous ?

$\endgroup$
  • $\begingroup$ Likelihood is a function of the (fixed but unknown) parameters given the observed data, so it's not a probability distribution. Integrating out some of those parameters doesn't make it one. $\endgroup$ – jbowman Dec 13 '17 at 3:15
2
$\begingroup$

Suppose we have the following joint probability density: \begin{equation} p(x,\theta,\alpha) %= p(x|\theta)\,p(\theta|\alpha)\,p(\alpha) , \end{equation} where $x = (x_1, \ldots, x_n)$. Because this is a probability density, we have \begin{equation} \iiint p(x,\theta,\alpha)\,dx\,d\theta\,d\alpha = 1 . \end{equation} Now suppose the joint density factors into the following three densities (two conditional densities and one marginal density): \begin{equation} p(x,\theta,\alpha) = p(x|\theta)\,p(\theta|\alpha)\,p(\alpha) . \end{equation} Given this factorization, we know \begin{align} \int p(x|\theta)\,dx &= 1 \\ \int p(\theta|\alpha)\,d\theta &= 1 \\ \int p(\alpha)\,d\alpha &= 1 . \end{align} Note that $p(x|\theta)$ is the likelihood for $\theta$ and $p(\theta|\alpha)$ is the likelihood for $\alpha$. As such, there is no requirement that either integrate to one or to any finite value for that matter: \begin{align} \int p(x|\theta)\,d\theta &= {}? \\ \int p(\theta|\alpha)\,d\alpha &= {}? . \end{align}

If we integrate out $\theta$ (not $\alpha$), we obtain \begin{equation} p(x|\alpha) = \int p(x|\theta)\,p(\theta|\alpha)\,d\theta , \end{equation} where \begin{equation} \int p(x|\alpha)\,dx = 1 . \end{equation} Note that $p(x|\alpha)$ is the marginal likelihood for $\alpha$. Again there is no requirement that it integrate to one or any finite value: \begin{equation} \int p(x|\alpha)\,d\alpha = {}? . \end{equation}

$\endgroup$
  • $\begingroup$ $p(x|\theta)$ is the likelihood for $\theta$ when $x$ is observed and $\theta$ is a random variable and the same symbol represents the conditional probability when $x$ is a Random variable and $\theta$ is given ? So there's a lot of overlapping symbology going on here and it's just a matter of being at loss with more symbolical descriptions ? $\endgroup$ –  redenzione11 Dec 13 '17 at 9:46
  • $\begingroup$ If we fix the value of $\theta$, then $p(x|\theta)$ is the conditional density for $x$. If we fix a value for $x$, then $p(x|\theta)$ is the likelihood for $\theta$. $\endgroup$ – mef Dec 13 '17 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.