I see these terms being used and I keep getting them mixed up. Is there a simple explanation of the differences between them?
$\begingroup$
$\endgroup$
asked Jul 26, 2010 at 9:12
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
7
The likelihood function usually depends on many parameters. Depending on the application, we are usually interested in only a subset of these parameters. For example, in linear regression, interest typically lies in the slope coefficients and not on the error variance.
Denote the parameters we are interested in as $\beta$ and the parameters that are not of primary interest as $\theta$. The standard way to approach the estimation problem is to maximize the likelihood function so that we obtain estimates of $\beta$ and $\theta$. However, since the primary interest lies in $\beta$ partial, profile and marginal likelihood offer alternative ways to estimate $\beta$ without estimating $\theta$.
In order to see the difference denote the standard likelihood by $L(\beta, \theta|\mathrm{data})$.
Maximum Likelihood
Find $\beta$ and $\theta$ that maximizes $L(\beta, \theta|\mathrm{data})$.
Partial Likelihood
If we can write the likelihood function as:
$$L(\beta, \theta|\mathrm{data}) = L_1(\beta|\mathrm{data}) L_2(\theta|\mathrm{data})$$
Then we simply maximize $L_1(\beta|\mathrm{data})$.
Profile Likelihood
If we can express $\theta$ as a function of $\beta$ then we replace $\theta$ with the corresponding function.
Say, $\theta = g(\beta)$. Then, we maximize:
$$L(\beta, g(\beta)|\mathrm{data})$$
Marginal Likelihood
We integrate out $\theta$ from the likelihood equation by exploiting the fact that we can identify the probability distribution of $\theta$ conditional on $\beta$.
-
2$\begingroup$ Note that the last definition here is an Integrated (or Bayesian) Likelihood, not a Marginal Likelihood. $\endgroup$– arsJul 31, 2010 at 0:09
-
$\begingroup$ Is this correct in the RHS for partial likelihood: "L2(θ|theta)"? $\endgroup$– jpalecekJul 31, 2010 at 0:13
-
$\begingroup$ @ars, would you please edit the answer and provide the definition of Marginal Likelihood then? $\endgroup$ Oct 3, 2016 at 17:06
-
1$\begingroup$ hmm, the standard way of defining likelihood is the probability of the DATA GIVEN the PARAMETERS, that is, the inverse conditioning of what's written here $\endgroup$ Jul 6, 2020 at 21:40
-
1$\begingroup$ @overdisperse In a Bayesian setting for example, we do have some prior on the parameters multiplied by the likelihood to obtain the posterior. Thus likelihood is the prob. of the data given the parameters, and posterior the reverse. Sure, data remains fixed and likelihood is considered as a function of the parameters used for optimization, but in terms of probabilities it is as described. Might be a question of convention though, depending on whether you want to emphasize the function part for optimization or the probabilistic situation $\endgroup$ Feb 21, 2022 at 0:23
$\begingroup$
$\endgroup$
All three are used when dealing with nuisance parameters in the completely specified likelihood function.
The marginal likelihood is the primary method to eliminate nuisance parameters in theory. It's a true likelihood function (i.e. it's proportional to the (marginal) probability of the observed data).
The partial likelihood is not a true likelihood in general. However, in some cases it can be treated as a likelihood for asymptotic inference. For example in Cox proportional hazards models, where it originated, we're interested in the observed rankings in the data (T1 > T2 > ..) without specifying the baseline hazard. Efron showed that the partial likelihood loses little to no information for a variety of hazard functions.
The profile likelihood is convenient when we have a multidimensional likelihood function and a single parameter of interest. It's specified by replacing the nuisance S by its MLE at each fixed T (the parameter of interest), i.e. L(T) = L(T, S(T)). This can work well in practice, though there is a potential bias in the MLE obtained in this way; the marginal likelihood corrects for this bias.
