EM Algorithm For Bipolar Normal Distribution

Question

Question: Let $x_1, \dots, x_m$ be an i.i.d. sample from a normal density with mean $\mu$ and variance $\sigma^2$. Suppose for each $x_i$ we observe $y_i = |x_i|$ . Formulate an EM algorithm for estimating $\mu$ and $\sigma^2$.

My solution:

Define a latent variable $Z$, when $z_i = 1, x_i = y_i$ and $z_i = 0, x_i = -y_i$ and the probability $p(z_i = 1| \Theta, y_i) = p$. It can be easily known that $-x_i \sim \mathcal{N}(-\mu, \sigma^2)$.

$$ \begin{equation} \begin{aligned} l(\mathbf{x}, \mathbf{z}, p, \Theta) = \sum_{i = 1}^m z_i\left[ -\frac{1}{2}\ln 2\pi - \frac{1}{2}\ln \sigma^2 -\frac{1}{2\sigma^2}(x_i - \mu)^2 + \ln p\right]\\ + \sum_{i = 1}^m (1 - z_i)\left[ -\frac{1}{2}\ln 2\pi - \frac{1}{2}\ln \sigma^2 -\frac{1}{2\sigma^2}(x_i + \mu)^2 + \ln (1-p)\right], \end{aligned} \end{equation}$$

The E step in EM algorithm is:$E_{\Theta_{n}}[l(\mathbf{x}, \mathbf{z}, p, \Theta) | \mathbf{y}]$.

My question:

1. It seems that some problems happens in my model since two latent variables $z_i, p$ and unknown $x_i$ involved in the E step. So could anyone tell me where is the mistake?

2. I see the answer for updating the $\mu$ involves $f(y_i | \Theta_n)$, but honestly speaking, from the E step: $E[x_iz_i | \Theta_n, y_i]$, there would be no $f_i$ involved. So how come the formula?

Thanks in advance!

---

The likelihood function can be further expressed as: \begin{equation} \begin{aligned} Q(\Theta, \Theta_{n}) = & E_{\Theta_{n}}[l(\mathbf{x}, \mathbf{z}, \Theta) | \mathbf{y}]\\ = & \sum_{i = 1}^m\left( -\frac{1}{2}\ln 2\pi - \frac{1}{2}\ln \sigma^2 - \frac{E_{\Theta_{n}}[x_i^2|y_i]}{2\sigma^2} - \frac{\mu^2}{2\sigma^2} - \frac{1-2\mu E_{\Theta_{n}}[x_iz_i|y_i]}{\sigma^2}\right) \end{aligned} \end{equation}

The expectation of $E[x_iz_i | \Theta_n, y_i]$ $$ \begin{equation} \begin{aligned} E[x z | \Theta_n, y] = & \int \sum_l xz_lp(x_k,z_l | \Theta_n, y) dx\\ = &\int xp(x_k,z = 1 | \Theta_n, y)dx\quad \text{only z = 1 left}\\ = & p(z = 1 | \Theta_n, y)\int x f(x | z = 1, \Theta_n, y)dx\\ = & \frac{f(y_i|\theta_n)}{f(y_i|\theta_n) + f(-y_i|\theta_n)} \mu_n \end{aligned} \end{equation}$$:

But still stuck.

Answer 1

1. There is no probability $p$ in this problem as $$\mathbb P_\theta(Z_i=1)=\mathbb P_\theta(X_i>0)=1-\Phi(\mu/\sigma)$$
2. There is only one type of latent variable, $\mathbf Z$, since $\mathbf X$ is a deterministic function of $\mathbf Y$ and $\mathbf Z$, as discussed below.
3. the complete likelihood can thus be expressed in terms of $\mathbf Y$ and $\mathbf Z$ only

If $X\sim\mathcal N(\mu,\sigma^2)$, then $Y=|X|$ has a Dirac mass distribution at $|X|$ conditional on $X$. The marginal distribution of $Y$ is the folded Normal, with density $$\sigma^{-1}\varphi(y;\mu,\sigma)+\sigma^{-1}\varphi(-y;\mu,\sigma)$$ Conversely, the distribution of $X$ conditional on $Y$ is a sum of Dirac masses at $Y$ and $-Y$ with respective masses proportional to $\varphi(y;\mu,\sigma)$ and $\varphi(-y;\mu,\sigma)$. Note that $$Z=\mathbb I_{X=|Y|}$$ is a deterministic transform of $(X,Y)$, hence that $Z$ is known given $(X,Y)$ and that $X$ is known given $(Z,Y)$. This implies that $$\mathbb E_{\theta_{n}}[l(\mathbf{X}, \mathbf{Z}, \theta) | \mathbf{y}] =\mathbb E_{\theta_{n}}[l(\mathbf{X(Z,Y)}, \mathbf{Z}, \theta) | \mathbf{y}] $$ and, since \begin{equation} \begin{aligned} l(\mathbf{x}, \mathbf{z}, p, \Theta) &= \sum_{i = 1}^m \mathbb I_{z_i=1}\left[ -\frac{1}{2}\ln 2\pi - \frac{1}{2}\ln \sigma^2 -\frac{1}{2\sigma^2}(x_i(1,y_i) - \mu)^2 \right]\\ &\quad + \sum_{i = 1}^m \mathbb I_{z_i=0}\left[ -\frac{1}{2}\ln 2\pi - \frac{1}{2}\ln \sigma^2 -\frac{1}{2\sigma^2}(x_i(0,y_i) - \mu)^2 \right],\\ &= \sum_{i = 1}^m \mathbb I_{z_i=1}\left[ -\frac{1}{2}\ln 2\pi - \frac{1}{2}\ln \sigma^2 -\frac{1}{2\sigma^2}(y_i - \mu)^2 \right]\\ &\quad + \sum_{i = 1}^m \mathbb I_{z_i=0}\left[ -\frac{1}{2}\ln 2\pi - \frac{1}{2}\ln \sigma^2 -\frac{1}{2\sigma^2}(-y_i - \mu)^2 \right], \end{aligned} \end{equation} the E-step writes as \begin{equation} \begin{aligned} \mathbb E_{\theta_n}[l(X,Z,\theta)|y) &= -\frac{m}{2}\ln 2\pi - \frac{m}{2}\ln \sigma^2- \frac{1}{2\sigma^2}\sum_{i = 1}^m \mathbb E_{\theta_n}[\mathbb I_{z_i=1}|y] (y_i - \mu)^2 \\ &\quad -\frac{1}{2\sigma^2} \sum_{i=1}^m \mathbb E_{\theta_n}[\mathbb I_{z_i=0}|y] (y_i + \mu)^2 \end{aligned} \end{equation} This implies that $\mu_{n+1}$ for the M-step is solution of the equation $$\sum_{i = 1}^m \mathbb E_{\theta_n}[\mathbb I_{z_i=1}|y] (\mu-y_i) +\sum_{i=1}^m \mathbb E_{\theta_n}[\mathbb I_{z_i=0}|y] (y_i + \mu) = 0$$