Relationship Between "Profile Likelihood" and "EM Algorithm"?

Question

I was reading Rao (2017) (Ch3) on profile likelihood. An example is provided which shows how the parameters of a Weibull Distribution can be estimated using the "profile likelihood approach":

It seems here what they are doing :

• They are able to re-arrange the original likelihood so that its now only in terms of "alpha". This re-arranged likelihood is now called the "profile likelihood"

• Then, they take the derivative of these re-arranged likelihood, set it to zero, and obtain a formula for the optimal value of "thetha" in terms of "alpha"

• Then, they go back to the original likelihood and replace "thetha" with this "optimal formula", and obtain an formula for the value optimal value of "alpha" based on thetha

Supposedly, using this approach (i.e. Profile Likelihood) can save on computational time - but I had the following question:

This procedure in some ways reminds of the EM Algorithm (different aspects of the likelihood function are sequentially maximized) - but in the case of the EM Algorithm, the EM algorithm is not guaranteed to provide parameter estimates that are "globally optimum". Thus, in the case of Profile Likelihood - do we know if:

1. The parameter estimates provided from maximizing the original likelihood will always be equal to the parameter estimates provided from maximizing the profile likelihoods?

2. If 1) is not true - do we expect that the estimates provided from one of these approaches will necessarily "dominate" the estimates provided from the other approach?

Thanks!

Note: I think the EM algorithm is used in situations when re-arranging the original likelihood in terms of the parameters is not always possible - but I am not sure about this.

Answer 1

The Expectation-Maximization algorithm is an optimization procedure that can be used to infer the parameters of a model, when observations depend on hidden latent variables that are not observed. For instance, if you have $T$ samples from a Weibull distribution (from which you want to infer the values of the parameters $\alpha$ and $\theta$), it would not make sense to use the EM algorithm, since observations do not depend on latent variables. A classical optimization algorithm (based either on the likelihood or on the profile likelihood) would be much simpler. On the other hand, if you have a Hidden Markov Model with hidden variable $x_t = f(x_{t-1})$ and where the observation is e.g. $y_t = x_t \cdot w$ (where $w$ is a random variable drawn from a Weibull distribution), then the EM is relevant to infer its $\alpha$ and $\theta$.

However, you are absolutely right to say that the EM algorithm can be interpreted as a form of coordinated ascent. It can be "seen as maximizing a joint function of the parameters and of the distribution over the unobserved variables [...] The E step maximizes this function with respect to the distribution over unobserved variables; the M step with respect to the parameters". This is detailed in the following seminal paper:

Neal, R. M., & Hinton, G. E. (1998). A view of the EM algorithm that justifies incremental, sparse, and other variants. In Learning in graphical models (pp. 355-368). Springer, Dordrecht.

Regarding your questions:

1. No, since the likelihood and the profile likelihoods are different functions. As you explained, in profile likelihood optimization, you set the value of one the parameters, use it to optimize the other parameter, and then go back to the first parameter. Which is not the same as solving $\nabla_{\alpha,\theta} \mathcal{L}(\alpha,\theta) = 0$ for $\alpha$ and $\theta$.
2. However, there is no formal and general theoretical link between the results of likelihood optimization and profile likelihood optimization. How they differ (and how they differ from the ground-truth parameters that we are trying to infer) will be a function of the model, of the value of the ground-truth parameters, and of the number of observations.

For instance, the classical way to compute the MLE for the parameters of a normal distribution $\mu$ and $\sigma$ implies to first compute the estimate of the mean $\hat{\mu}$ and then to use it to infer $\hat{\sigma}$. This can be seen as a form of profile likelihood optimization.

For a Weibull distribution, the effect of the profile likelihood approximation might depend on $T$; intuitively, for a very small number of observations, both methods should give results that are equally bad. In the absence of theoretical results, the best way to quantify the difference between likelihood and profile likelihood approaches is to use MCMC simulations (i.e. for a given $T$ and ground-truth $\alpha^*$ and $\theta^*$, generate several independent sets of $T$ samples, and apply both methods to each of them to systematically compare their results).

Answer 2

The profile likelihood approach is essentially just splitting a multivariate optimisation into two sequential parts. Given a likelihood function $\mathcal{L}_\mathbf{x}: \mathcal{A} \times \Theta \rightarrow \mathbb{R}_+$ we can write the maximum as:

$$\max_{\alpha \in \mathcal{A} \\ \theta \in \Theta} \mathcal{L}_\mathbf{x}(\alpha, \theta) = \max_{\alpha \in \mathcal{A}} \mathcal{L}_\mathbf{x}(\alpha, \hat{\theta}(\alpha)),$$

where we use the conditional maximiser:

$$\hat{\theta}(\alpha) = \underset{\theta \in \Theta}{\text{arg max}} \ \mathcal{L}_\mathbf{x}(\alpha, \theta) \quad \quad \quad \quad \quad \text{for all } \alpha \in \mathcal{A}.$$

The function $\alpha \mapsto \mathcal{L}_\mathbf{x}(\alpha, \hat{\theta}(\alpha))$ in the former expression is the profile likelihood function, and it depends on a previous step of finding the conditional maximiser for the nuisance parameter $\theta$. So long as the relevant maxima exist, the profile likelihood method will always properly define the MLE.

The profile likelihood approach differs from the EM algorithm, but they have some heuristic similarities. In the profile likelihood method the goal is to split the optimisation based on two sets of parameter values whereby we conduct a conditional optimisation in one parameter and then conduct the marginal optimisation in the other. In the EM algorithm we are instead concerned with problems where the goal is to split the optimisation based on two sets of sample vectors, one of which is an unobserved "latent variable". This latter algorithm is used in cases where the marginal likelihood (integrating out the latent variable) is intractable, but the full likelihood is simpler.