Question: Let $x_1, \dots, x_m$ be an i.i.d. sample from a normal density with mean $\mu$ and variance $\sigma^2$. Suppose for each $x_i$ we observe $y_i = |x_i|$ . Formulate an EM algorithm for estimating $\mu$ and $\sigma^2$.

My solution:

Define a latent variable $Z$, when $z_i = 1, x_i = y_i$ and $z_i = 0, x_i = -y_i$ and the probability $p(z_i = 1| \Theta, y_i) = p$. It can be easily known that $-x_i \sim \mathcal{N}(-\mu, \sigma^2)$.

$$ \begin{equation} \begin{aligned} l(\mathbf{x}, \mathbf{z}, p, \Theta) = \sum_{i = 1}^m z_i\left[ -\frac{1}{2}\ln 2\pi - \frac{1}{2}\ln \sigma^2 -\frac{1}{2\sigma^2}(x_i - \mu)^2 + \ln p\right]\\ + \sum_{i = 1}^m (1 - z_i)\left[ -\frac{1}{2}\ln 2\pi - \frac{1}{2}\ln \sigma^2 -\frac{1}{2\sigma^2}(x_i + \mu)^2 + \ln (1-p)\right], \end{aligned} \end{equation}$$

The E step in EM algorithm is:$E_{\Theta_{n}}[l(\mathbf{x}, \mathbf{z}, p, \Theta) | \mathbf{y}]$.

My question:

  1. It seems that some problems happens in my model since two latent variables $z_i, p$ and unknown $x_i$ involved in the E step. So could anyone tell me where is the mistake?

  2. I see the answer for updating the $\mu$ involves $f(y_i | \Theta_n)$, but honestly speaking, from the E step: $E[x_iz_i | \Theta_n, y_i]$, there would be no $f_i$ involved. So how come the formula?

Thanks in advance!

The likelihood function can be further expressed as: \begin{equation} \begin{aligned} Q(\Theta, \Theta_{n}) = & E_{\Theta_{n}}[l(\mathbf{x}, \mathbf{z}, \Theta) | \mathbf{y}]\\ = & \sum_{i = 1}^m\left( -\frac{1}{2}\ln 2\pi - \frac{1}{2}\ln \sigma^2 - \frac{E_{\Theta_{n}}[x_i^2|y_i]}{2\sigma^2} - \frac{\mu^2}{2\sigma^2} - \frac{1-2\mu E_{\Theta_{n}}[x_iz_i|y_i]}{\sigma^2}\right) \end{aligned} \end{equation}

The expectation of $E[x_iz_i | \Theta_n, y_i]$ $$ \begin{equation} \begin{aligned} E[x z | \Theta_n, y] = & \int \sum_l xz_lp(x_k,z_l | \Theta_n, y) dx\\ = &\int xp(x_k,z = 1 | \Theta_n, y)dx\quad \text{only z = 1 left}\\ = & p(z = 1 | \Theta_n, y)\int x f(x | z = 1, \Theta_n, y)dx\\ = & \frac{f(y_i|\theta_n)}{f(y_i|\theta_n) + f(-y_i|\theta_n)} \mu_n \end{aligned} \end{equation}$$:

But still stuck.

  1. There is no probability $p$ in this problem as $$\mathbb P_\theta(Z_i=1)=\mathbb P_\theta(X_i>0)=1-\Phi(\mu/\sigma)$$
  2. There is only one type of latent variable, $\mathbf Z$, since $\mathbf X$ is a deterministic function of $\mathbf Y$ and $\mathbf Z$, as discussed below.
  3. the complete likelihood can thus be expressed in terms of $\mathbf Y$ and $\mathbf Z$ only

If $X\sim\mathcal N(\mu,\sigma^2)$, then $Y=|X|$ has a Dirac mass distribution at $|X|$ conditional on $X$. The marginal distribution of $Y$ is the folded Normal, with density $$\sigma^{-1}\varphi(y;\mu,\sigma)+\sigma^{-1}\varphi(-y;\mu,\sigma)$$ Conversely, the distribution of $X$ conditional on $Y$ is a sum of Dirac masses at $Y$ and $-Y$ with respective masses proportional to $\varphi(y;\mu,\sigma)$ and $\varphi(-y;\mu,\sigma)$. Note that $$Z=\mathbb I_{X=|Y|}$$ is a deterministic transform of $(X,Y)$, hence that $Z$ is known given $(X,Y)$ and that $X$ is known given $(Z,Y)$. This implies that $$\mathbb E_{\theta_{n}}[l(\mathbf{X}, \mathbf{Z}, \theta) | \mathbf{y}] =\mathbb E_{\theta_{n}}[l(\mathbf{X(Z,Y)}, \mathbf{Z}, \theta) | \mathbf{y}] $$ and, since \begin{equation} \begin{aligned} l(\mathbf{x}, \mathbf{z}, p, \Theta) &= \sum_{i = 1}^m \mathbb I_{z_i=1}\left[ -\frac{1}{2}\ln 2\pi - \frac{1}{2}\ln \sigma^2 -\frac{1}{2\sigma^2}(x_i(1,y_i) - \mu)^2 \right]\\ &\quad + \sum_{i = 1}^m \mathbb I_{z_i=0}\left[ -\frac{1}{2}\ln 2\pi - \frac{1}{2}\ln \sigma^2 -\frac{1}{2\sigma^2}(x_i(0,y_i) - \mu)^2 \right],\\ &= \sum_{i = 1}^m \mathbb I_{z_i=1}\left[ -\frac{1}{2}\ln 2\pi - \frac{1}{2}\ln \sigma^2 -\frac{1}{2\sigma^2}(y_i - \mu)^2 \right]\\ &\quad + \sum_{i = 1}^m \mathbb I_{z_i=0}\left[ -\frac{1}{2}\ln 2\pi - \frac{1}{2}\ln \sigma^2 -\frac{1}{2\sigma^2}(-y_i - \mu)^2 \right], \end{aligned} \end{equation} the E-step writes as \begin{equation} \begin{aligned} \mathbb E_{\theta_n}[l(X,Z,\theta)|y) &= -\frac{m}{2}\ln 2\pi - \frac{m}{2}\ln \sigma^2- \frac{1}{2\sigma^2}\sum_{i = 1}^m \mathbb E_{\theta_n}[\mathbb I_{z_i=1}|y] (y_i - \mu)^2 \\ &\quad -\frac{1}{2\sigma^2} \sum_{i=1}^m \mathbb E_{\theta_n}[\mathbb I_{z_i=0}|y] (y_i + \mu)^2 \end{aligned} \end{equation} This implies that $\mu_{n+1}$ for the M-step is solution of the equation $$\sum_{i = 1}^m \mathbb E_{\theta_n}[\mathbb I_{z_i=1}|y] (\mu-y_i) +\sum_{i=1}^m \mathbb E_{\theta_n}[\mathbb I_{z_i=0}|y] (y_i + \mu) = 0$$

  • $\begingroup$ Thanks! Good answer! $\endgroup$ Oct 23 '20 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.