# EM Algorithm For Bipolar Normal Distribution

Question: Let $$x_1, \dots, x_m$$ be an i.i.d. sample from a normal density with mean $$\mu$$ and variance $$\sigma^2$$. Suppose for each $$x_i$$ we observe $$y_i = |x_i|$$ . Formulate an EM algorithm for estimating $$\mu$$ and $$\sigma^2$$.

My solution:

Define a latent variable $$Z$$, when $$z_i = 1, x_i = y_i$$ and $$z_i = 0, x_i = -y_i$$ and the probability $$p(z_i = 1| \Theta, y_i) = p$$. It can be easily known that $$-x_i \sim \mathcal{N}(-\mu, \sigma^2)$$.

\begin{equation} \begin{aligned} l(\mathbf{x}, \mathbf{z}, p, \Theta) = \sum_{i = 1}^m z_i\left[ -\frac{1}{2}\ln 2\pi - \frac{1}{2}\ln \sigma^2 -\frac{1}{2\sigma^2}(x_i - \mu)^2 + \ln p\right]\\ + \sum_{i = 1}^m (1 - z_i)\left[ -\frac{1}{2}\ln 2\pi - \frac{1}{2}\ln \sigma^2 -\frac{1}{2\sigma^2}(x_i + \mu)^2 + \ln (1-p)\right], \end{aligned} \end{equation}

The E step in EM algorithm is:$$E_{\Theta_{n}}[l(\mathbf{x}, \mathbf{z}, p, \Theta) | \mathbf{y}]$$.

My question:

1. It seems that some problems happens in my model since two latent variables $$z_i, p$$ and unknown $$x_i$$ involved in the E step. So could anyone tell me where is the mistake?

2. I see the answer for updating the $$\mu$$ involves $$f(y_i | \Theta_n)$$, but honestly speaking, from the E step: $$E[x_iz_i | \Theta_n, y_i]$$, there would be no $$f_i$$ involved. So how come the formula?

The likelihood function can be further expressed as: \begin{equation} \begin{aligned} Q(\Theta, \Theta_{n}) = & E_{\Theta_{n}}[l(\mathbf{x}, \mathbf{z}, \Theta) | \mathbf{y}]\\ = & \sum_{i = 1}^m\left( -\frac{1}{2}\ln 2\pi - \frac{1}{2}\ln \sigma^2 - \frac{E_{\Theta_{n}}[x_i^2|y_i]}{2\sigma^2} - \frac{\mu^2}{2\sigma^2} - \frac{1-2\mu E_{\Theta_{n}}[x_iz_i|y_i]}{\sigma^2}\right) \end{aligned} \end{equation}

The expectation of $$E[x_iz_i | \Theta_n, y_i]$$ \begin{equation} \begin{aligned} E[x z | \Theta_n, y] = & \int \sum_l xz_lp(x_k,z_l | \Theta_n, y) dx\\ = &\int xp(x_k,z = 1 | \Theta_n, y)dx\quad \text{only z = 1 left}\\ = & p(z = 1 | \Theta_n, y)\int x f(x | z = 1, \Theta_n, y)dx\\ = & \frac{f(y_i|\theta_n)}{f(y_i|\theta_n) + f(-y_i|\theta_n)} \mu_n \end{aligned} \end{equation}:

But still stuck.

1. There is no probability $$p$$ in this problem as $$\mathbb P_\theta(Z_i=1)=\mathbb P_\theta(X_i>0)=1-\Phi(\mu/\sigma)$$
2. There is only one type of latent variable, $$\mathbf Z$$, since $$\mathbf X$$ is a deterministic function of $$\mathbf Y$$ and $$\mathbf Z$$, as discussed below.
3. the complete likelihood can thus be expressed in terms of $$\mathbf Y$$ and $$\mathbf Z$$ only
If $$X\sim\mathcal N(\mu,\sigma^2)$$, then $$Y=|X|$$ has a Dirac mass distribution at $$|X|$$ conditional on $$X$$. The marginal distribution of $$Y$$ is the folded Normal, with density $$\sigma^{-1}\varphi(y;\mu,\sigma)+\sigma^{-1}\varphi(-y;\mu,\sigma)$$ Conversely, the distribution of $$X$$ conditional on $$Y$$ is a sum of Dirac masses at $$Y$$ and $$-Y$$ with respective masses proportional to $$\varphi(y;\mu,\sigma)$$ and $$\varphi(-y;\mu,\sigma)$$. Note that $$Z=\mathbb I_{X=|Y|}$$ is a deterministic transform of $$(X,Y)$$, hence that $$Z$$ is known given $$(X,Y)$$ and that $$X$$ is known given $$(Z,Y)$$. This implies that $$\mathbb E_{\theta_{n}}[l(\mathbf{X}, \mathbf{Z}, \theta) | \mathbf{y}] =\mathbb E_{\theta_{n}}[l(\mathbf{X(Z,Y)}, \mathbf{Z}, \theta) | \mathbf{y}]$$ and, since \begin{equation} \begin{aligned} l(\mathbf{x}, \mathbf{z}, p, \Theta) &= \sum_{i = 1}^m \mathbb I_{z_i=1}\left[ -\frac{1}{2}\ln 2\pi - \frac{1}{2}\ln \sigma^2 -\frac{1}{2\sigma^2}(x_i(1,y_i) - \mu)^2 \right]\\ &\quad + \sum_{i = 1}^m \mathbb I_{z_i=0}\left[ -\frac{1}{2}\ln 2\pi - \frac{1}{2}\ln \sigma^2 -\frac{1}{2\sigma^2}(x_i(0,y_i) - \mu)^2 \right],\\ &= \sum_{i = 1}^m \mathbb I_{z_i=1}\left[ -\frac{1}{2}\ln 2\pi - \frac{1}{2}\ln \sigma^2 -\frac{1}{2\sigma^2}(y_i - \mu)^2 \right]\\ &\quad + \sum_{i = 1}^m \mathbb I_{z_i=0}\left[ -\frac{1}{2}\ln 2\pi - \frac{1}{2}\ln \sigma^2 -\frac{1}{2\sigma^2}(-y_i - \mu)^2 \right], \end{aligned} \end{equation} the E-step writes as \begin{equation} \begin{aligned} \mathbb E_{\theta_n}[l(X,Z,\theta)|y) &= -\frac{m}{2}\ln 2\pi - \frac{m}{2}\ln \sigma^2- \frac{1}{2\sigma^2}\sum_{i = 1}^m \mathbb E_{\theta_n}[\mathbb I_{z_i=1}|y] (y_i - \mu)^2 \\ &\quad -\frac{1}{2\sigma^2} \sum_{i=1}^m \mathbb E_{\theta_n}[\mathbb I_{z_i=0}|y] (y_i + \mu)^2 \end{aligned} \end{equation} This implies that $$\mu_{n+1}$$ for the M-step is solution of the equation $$\sum_{i = 1}^m \mathbb E_{\theta_n}[\mathbb I_{z_i=1}|y] (\mu-y_i) +\sum_{i=1}^m \mathbb E_{\theta_n}[\mathbb I_{z_i=0}|y] (y_i + \mu) = 0$$