# Marginalizing over a parameter: integrate the total joint likelihood, or the each individual likelihood?

This is a general question about a model-fitting task. Suppose you have IID data $Y_1, ..., Y_n$ arising from a data generating modeling indexed by a parameter $(\theta, \lambda)$, where you are only interested in $\theta$.

Under my model, I can easily calculate $P(Y_i | \theta, \lambda)$ for each $i$, and thus the likelihood of the sample is

$$P(Y_1, ..., Y_n | \theta, \lambda) = \prod_{i=1}^{n} P(Y_i | \theta, \lambda)$$

I want to integrate $\lambda$ out of this expression. Is the correct expression for the marginal likelihood then

$$L(\theta) = \int_{\Lambda} \ \prod_{i=1}^{n} P(Y_i | \theta, \lambda) \ d \lambda \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ (1)$$

or do you calculate the marginal likelihood for each subject, and then take the product:

$$L(\theta) = \prod_{i=1}^{n} L_i (\theta) = \prod_{i=1}^{n} \int_{\Lambda} P(Y_i | \theta, \lambda) \ d \lambda \ \ \ \ \ \ \ (2)$$

Both seem reasonable but they obviously are not equal. Controlling numerical underflow is obviously more difficult with expression (1), but I'm mainly concerned with which answer obeys convention.

Note: I am aware this is not exactly a "likelihood function", rather marginalizing over part of a posterior distribution with an uninformative prior.

First off, neither (1) or (2) are quite right because you didn't include the prior on $\lambda$ in the integrand [call these revised integrals (1') and (2'), respectively]. Just remember that marginalization occurs with respect to a joint distribution: $p(x) = \int p(x,y) \,dy = \int p(x \mid y) p(y) \,dy$. So (1') looks like $$L(\theta) = p(\mathbf{Y} \mid \theta) = \int p(\mathbf{Y} \mid \theta, \lambda)\, p(\lambda) \,d\lambda =\int \left[ \prod_{i=1}^n p(Y_i \mid \theta, \lambda) \right] p(\lambda) \,d\lambda.$$
Second, if you want to calculate $P(\mathbf{Y} \mid \theta)$ then you should go for (1') because it is always true. (2') is false unless you assume that the outcomes are independent of $\lambda$ given $\theta$. Of course, that means that $p(Y_i \mid \theta, \lambda) = p(Y_i \mid \theta)$, which I doubt is what you want to assume in general.
Third, for posterior inference on $\theta$ it may be easier to just marginalize out $\lambda$ from the joint posterior of both parameters: $p(\theta \mid \mathbf{Y}) = \int p(\theta, \lambda \mid \mathbf{Y}) \, d\lambda$, but of course that depends on you and what problem you're working on.
• Thank you Matt. I was imagining an "uniform integrated likelihood", so using a constant prior for $\lambda$. Regarding your comment about (2), I don't understand why that is equivalent to assuming $P(Y|\theta, \lambda) = P(Y|\theta)$. Isn't it true that "integrated likelihood" for subject $i$ is $\int P(Y_i | \theta, \lambda) d \lambda$? Are you suggesting that by integrating out $\lambda$, I'm inducing some kind of dependence between the $Y$s, thereby making it incorrect to take the product? Feb 26, 2016 at 23:46
• @linksys The other $Y$s have information about $\lambda$ within them, so unless the $Y$ are only dependent on $\theta$ and not $\lambda$, then you do actually induce dependence in the $Y$ by marginalizing out $\lambda$. (2) is not equal to the likelihood $P(\theta \mid \bf{Y})$ without that conditional independence of $\lambda$ assumption. To be specific, the product of $P(Y_i \mid \theta)$s is $P(Y_1, \dots, Y_n \mid \theta)$ only when the $Y$s are independent conditional on $\theta$, which is the definition of conditional independence.