60
$\begingroup$

I see these terms being used and I keep getting them mixed up. Is there a simple explanation of the differences between them?

$\endgroup$
62
$\begingroup$

The likelihood function usually depends on many parameters. Depending on the application, we are usually interested in only a subset of these parameters. For example, in linear regression, interest typically lies in the slope coefficients and not on the error variance.

Denote the parameters we are interested in as $\beta$ and the parameters that are not of primary interest as $\theta$. The standard way to approach the estimation problem is to maximize the likelihood function so that we obtain estimates of $\beta$ and $\theta$. However, since the primary interest lies in $\beta$ partial, profile and marginal likelihood offer alternative ways to estimate $\beta$ without estimating $\theta$.

In order to see the difference denote the standard likelihood by $L(\beta, \theta|\mathrm{data})$.

Maximum Likelihood

Find $\beta$ and $\theta$ that maximizes $L(\beta, \theta|\mathrm{data})$.

Partial Likelihood

If we can write the likelihood function as:

$$L(\beta, \theta|\mathrm{data}) = L_1(\beta|\mathrm{data}) L_2(\theta|\mathrm{data})$$

Then we simply maximize $L_1(\beta|\mathrm{data})$.

Profile Likelihood

If we can express $\theta$ as a function of $\beta$ then we replace $\theta$ with the corresponding function.

Say, $\theta = g(\beta)$. Then, we maximize:

$$L(\beta, g(\beta)|\mathrm{data})$$

Marginal Likelihood

We integrate out $\theta$ from the likelihood equation by exploiting the fact that we can identify the probability distribution of $\theta$ conditional on $\beta$.

$\endgroup$
  • 2
    $\begingroup$ Note that the last definition here is an Integrated (or Bayesian) Likelihood, not a Marginal Likelihood. $\endgroup$ – ars Jul 31 '10 at 0:09
  • $\begingroup$ Is this correct in the RHS for partial likelihood: "L2(θ|theta)"? $\endgroup$ – jpalecek Jul 31 '10 at 0:13
  • $\begingroup$ @ars, would you please edit the answer and provide the definition of Marginal Likelihood then? $\endgroup$ – Waldir Leoncio Oct 3 '16 at 17:06
13
$\begingroup$

All three are used when dealing with nuisance parameters in the completely specified likelihood function.

The marginal likelihood is the primary method to eliminate nuisance parameters in theory. It's a true likelihood function (i.e. it's proportional to the (marginal) probability of the observed data).

The partial likelihood is not a true likelihood in general. However, in some cases it can be treated as a likelihood for asymptotic inference. For example in Cox proportional hazards models, where it originated, we're interested in the observed rankings in the data (T1 > T2 > ..) without specifying the baseline hazard. Efron showed that the partial likelihood loses little to no information for a variety of hazard functions.

The profile likelihood is convenient when we have a multidimensional likelihood function and a single parameter of interest. It's specified by replacing the nuisance S by its MLE at each fixed T (the parameter of interest), i.e. L(T) = L(T, S(T)). This can work well in practice, though there is a potential bias in the MLE obtained in this way; the marginal likelihood corrects for this bias.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.