While I am reading ESL, I have some questions in chapter 8 (Model inference and averaging). Specifically,8.5.2 The EM algorithm in general. This part explains how EM works, in general, referring to the previous example of the two-component Gaussian mixture. the notations for two-component Gaussian mixture example: $Y_1 ∼ N(μ_1,σ_1^2) , Y_2 ∼ N(μ_2,σ_2^2), Y = (1−\Delta)·Y_1 +\Delta·Y_2$,

$\phi_{\theta}(x)$ - denoting normal density

$g_Y (y) = (1 − \pi) \phi_{\theta_1} (y) + \pi\phi_{\theta_2} (y)$ - density of $Y$

$\theta = (\pi, \theta_1, \theta_2) = (\pi, \mu_1, \sigma_1^2, \mu_2, \sigma_2^2)$ - parameters

$l(θ; Z) = \sum_{i=1}^N log[(1-\pi)\phi_{\theta_1}(y_i) + \pi\phi_{\theta_2}(y_i)]$ - log-likelihood based on N traing cases

$\Delta_i$ - unobserved latent variables taking values 0 or 1; if $\Delta_i = 1$ then $Y_i$ comes from model 2.

Supposing we knew the values of $\Delta_i$'s (8.40): $l_0(θ; Z, \Delta) = \sum_{i=1}^N [(1-\Delta_i)\phi_{\theta_1}(y_i) + \Delta_i\phi_{\theta_2}(y_i)] + \sum_{i=1}^N [(1-\Delta_i)log(1-\pi) + \Delta_ilog\pi]$

substitution for each $\Delta_i$ in (8.40) with its expected value (8.41):

$\gamma_i(\theta) = E(\Delta_i|\theta, Z) = Pr(\Delta_i = 1|\theta,Z)$

Algorithm 8.2 gives the general formulation of the EM algorithm. Our observed data is $Z$, having log-likelihood $l(θ; Z)$ depending on parameters $θ$. The latent or missing data is $Z^m$, so that the complete data is $T = (Z, Z^m)$ with log-likelihood $l_0(θ; T)$, $l_0$ based on the complete density. In the mixture problem $(Z, Z^m) = (y, Δ)$, and $l_0(θ; T)$ is given in (8.40). In our mixture example, $E(l_0(θ′; T)|Z, θˆ(j))$ is simply (8.40) with the $Δ_i$ replaced by the responsibilities $\hat{\gamma}_i(\hat{\theta})$, and the maximizers in step 3 are just weighted means and variances.

What does the bold line mean? I think I am not sure how it takes the expectation of the log-likelihood and ending up replacing $\Delta_i$ with $\hat{\gamma}_i(\hat{\theta})$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.