Derivation of a marginal log likelihood

Question

I have a question concerning a derivation of a marginal log-Likelihood function that I found in a paper and that I do not understand. It's pretty basic so sorry for this in advance.

Assume that the marginal Likelihood of object $i$ is given by

$f(e_i) = \int_{\theta} f(e_i|\theta)g(\theta)\, d\theta$

and the Likelihood for $N$ objects is $L = \Pi_{i = 1}^{N} f(e_i)$. Set $f(e_i) = f_i, c_i = f(e_i|\theta)$, and $g(\theta) = g$.

We are interested in estimating parameters that are in $c_i$. It may be important that $\theta$ is a two-dimensional vector. We therefore consider the log-Likelihood given by

$\ln L = \sum_{i=1}^{N} \ln(f_i) = \sum_{i=1}^{N} \ln \left[ \int_{\theta} (c_i \cdot g )\,d\theta \right]$

In the paper they say that the first derivative of $\ln L$ concerning a parameter vector $\eta$ is given by

$\frac{\partial \ln L}{\partial \eta} = \sum_{i=1}^{N} \frac{1}{f_i} \int_{\theta} \left[ \frac{\partial \ln(f_i)}{\partial \eta} \cdot f_i \cdot g \right] \, d\theta$

My question is how one arrives at this derivation. The first part of the derivation, i.e., $\frac{1}{f_i}$ seems to be an application of the chain rule. But in this case one would also need the derivation of

$\frac{\partial }{\partial \eta} \int_{\theta} f(e_i|\theta)g(\theta) \, d\theta$ which does not seem to involve another logarithm so I would not end up with $\frac{\partial \ln(f_i)}{\partial \eta}$ within the integral. Are the authors using another rule? I would be happy, if you could give me a hint.

Answer 1

Let's try to clear up some of the notation and simplify the problem. We'll return to your initial notation, which is a lot clearer, and extend it by showing where $\eta$ fits in:

$$\ln L = \Sigma_i \ln f(e_i|\eta) = \Sigma_i \ln \int_{\theta}f(e_i|\eta, \theta)g(\theta)d\theta$$

I'm making the simplifying assumption that $\eta$ is a parameter of the conditional distribution.

Now, a bit of notational simplification. Note that the sum over $i = 1\dots N$ doesn't matter; if you have the right expression for $i=1$, you'll have it right for all the other values of $i$ too, because they are all the same except for the index. So we can drop the summation and index altogether.

As written, the equation which is causing you trouble doesn't seem very useful, and I suspect you actually meant something else. What you have written translates into:

$$\frac{\partial \ln L}{\partial \eta} = \frac{1}{f(e|\eta)} \int_{\theta}\frac{\partial \ln f(e|\eta)}{\partial \eta}f(e|\eta)g(\theta)d\theta$$

Observe that the first two terms under the integral are not functions of $\theta$. They can therefore be moved outside the integral, giving:

$$\frac{\partial \ln L}{\partial \eta} = \frac{1}{f(e|\eta)} \frac{\partial \ln f(e|\eta)}{\partial \eta}f(e|\eta)\int_{\theta}g(\theta)d\theta$$

Making the obvious cancellation and observing that $\int_{\theta}g(\theta)d\theta = 1$ enables us to simplify the above:

$$\frac{\partial \ln L}{\partial \eta} = \frac{\partial \ln f(e|\eta)}{\partial \eta}$$

which would hardly seem worth the effort of expanding to the original form.

On to the problem that I think you are trying to understand. We start with the derivative, which can be written as:

$$\frac{\partial \ln L}{\partial \eta} = \frac{\partial \ln \int_{\theta}f(e|\eta, \theta)g(\theta)d\theta}{\partial \eta}$$

Using the chain rule gives:

$$\frac{1}{\int_{\theta}f(e|\eta, \theta)g(\theta)d\theta} \frac{\partial \int_{\theta}f(e|\eta, \theta)g(\theta)d\theta}{\partial \eta} $$

Substituting $f(e|\theta)$ for the integral in the denominator of the first term and reversing the order of differentiation and integration results in:

$$\frac{1}{f(e|\eta)}\int_{\theta}\frac{\partial f(e|\eta, \theta)}{\partial \eta}g(\theta)d\theta$$

Now for a minor sleight of hand. We observe that:

$$\frac{\partial f(e|\eta, \theta)}{\partial \eta} = \frac{\partial \ln f(e|\eta, \theta)}{\partial \eta} f(e|\eta, \theta)$$

To see this, just divide both sides by $f(e|\eta, \theta)$. Substitution gives us:

$$\frac{\partial \ln L}{\partial \eta} = \frac{1}{f(e|\eta)}\int_{\theta}\frac{\partial \ln f(e|\eta, \theta)}{\partial \eta} f(e|\eta, \theta)g(\theta)d\theta$$

which is what I think you may have meant to write.