1
$\begingroup$

This is a general question about a model-fitting task. Suppose you have IID data $Y_1, ..., Y_n$ arising from a data generating modeling indexed by a parameter $(\theta, \lambda)$, where you are only interested in $\theta$.

Under my model, I can easily calculate $P(Y_i | \theta, \lambda)$ for each $i$, and thus the likelihood of the sample is

$$ P(Y_1, ..., Y_n | \theta, \lambda) = \prod_{i=1}^{n} P(Y_i | \theta, \lambda) $$

I want to integrate $\lambda$ out of this expression. Is the correct expression for the marginal likelihood then

$$ L(\theta) = \int_{\Lambda} \ \prod_{i=1}^{n} P(Y_i | \theta, \lambda) \ d \lambda \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ (1) $$

or do you calculate the marginal likelihood for each subject, and then take the product:

$$ L(\theta) = \prod_{i=1}^{n} L_i (\theta) = \prod_{i=1}^{n} \int_{\Lambda} P(Y_i | \theta, \lambda) \ d \lambda \ \ \ \ \ \ \ (2)$$

Both seem reasonable but they obviously are not equal. Controlling numerical underflow is obviously more difficult with expression (1), but I'm mainly concerned with which answer obeys convention.

Note: I am aware this is not exactly a "likelihood function", rather marginalizing over part of a posterior distribution with an uninformative prior.

$\endgroup$
3
$\begingroup$

First off, neither (1) or (2) are quite right because you didn't include the prior on $\lambda$ in the integrand [call these revised integrals (1') and (2'), respectively]. Just remember that marginalization occurs with respect to a joint distribution: $p(x) = \int p(x,y) \,dy = \int p(x \mid y) p(y) \,dy$. So (1') looks like $$ L(\theta) = p(\mathbf{Y} \mid \theta) = \int p(\mathbf{Y} \mid \theta, \lambda)\, p(\lambda) \,d\lambda =\int \left[ \prod_{i=1}^n p(Y_i \mid \theta, \lambda) \right] p(\lambda) \,d\lambda. $$

Second, if you want to calculate $P(\mathbf{Y} \mid \theta)$ then you should go for (1') because it is always true. (2') is false unless you assume that the outcomes are independent of $\lambda$ given $\theta$. Of course, that means that $p(Y_i \mid \theta, \lambda) = p(Y_i \mid \theta)$, which I doubt is what you want to assume in general.

Third, for posterior inference on $\theta$ it may be easier to just marginalize out $\lambda$ from the joint posterior of both parameters: $p(\theta \mid \mathbf{Y}) = \int p(\theta, \lambda \mid \mathbf{Y}) \, d\lambda$, but of course that depends on you and what problem you're working on.

$\endgroup$
  • $\begingroup$ Thank you Matt. I was imagining an "uniform integrated likelihood", so using a constant prior for $\lambda$. Regarding your comment about (2), I don't understand why that is equivalent to assuming $P(Y|\theta, \lambda) = P(Y|\theta)$. Isn't it true that "integrated likelihood" for subject $i$ is $\int P(Y_i | \theta, \lambda) d \lambda$? Are you suggesting that by integrating out $\lambda$, I'm inducing some kind of dependence between the $Y$s, thereby making it incorrect to take the product? $\endgroup$ – linksys Feb 26 '16 at 23:46
  • $\begingroup$ @linksys The other $Y$s have information about $\lambda$ within them, so unless the $Y$ are only dependent on $\theta$ and not $\lambda$, then you do actually induce dependence in the $Y$ by marginalizing out $\lambda$. (2) is not equal to the likelihood $P(\theta \mid \bf{Y})$ without that conditional independence of $\lambda$ assumption. To be specific, the product of $P(Y_i \mid \theta)$s is $P(Y_1, \dots, Y_n \mid \theta)$ only when the $Y$s are independent conditional on $\theta$, which is the definition of conditional independence. $\endgroup$ – user44764 Feb 27 '16 at 0:40
  • $\begingroup$ I understand now. I will certainly use (1). Thanks very much! $\endgroup$ – linksys Feb 27 '16 at 1:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.