1
$\begingroup$

In this calculation

https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm#E_step

a probability, $P(Z_i = j|X_i=x_i;\theta^{(t)})$ is evaluated using bayes theorem, and then each of the probabilites is written as densities (as opposed to the integrals of densities, which probabilites are).

I should clarify. We have

$$P(Z_i = j|X_i=x_i;\theta^{(t)}) = \frac{P(X_i=x_i|Z_i = 1,\theta^{(t)})P(Z_i = 1)}{P(X_i|\theta^{(t)})} = \frac{f(x_i,\mu_j,\Sigma_j) \tau_j}{f(x_i,\mu_1,\Sigma_1) \tau_1 + f(x_i,\mu_2,\Sigma_2) \tau_2} $$

which implies that $$P(X_i=x_i|Z_i = 1,\theta^{(t)}) = f(x_i,\mu_j,\Sigma_j)$$ I feel like it should rather be $$P(X_i=x_i|Z_i = 1,\theta^{(t)}) = \int_{-\infty}^{x_i} f(s,\mu_j,\Sigma_j)ds$$

So why is the probability equal to the density, and not the integral of the density?

$\endgroup$
0

1 Answer 1

3
$\begingroup$

Since the OP is refering to the full E-formula from Wikipedia, here is the E-step described in Wikipedia for a mixture of two Gaussian distributions (with a typo on the side):

\begin{align}Q(\theta|\theta^{(t)}) &= \operatorname{E}_{\mathbf{Z}|\mathbf{X},\mathbf{\theta}^{(t)}} [\log L(\theta;\mathbf{x},\mathbf{Z}) ] \\ &= \operatorname{E}_{\mathbf{Z}|\mathbf{X},\mathbf{\theta}^{(t)}} [\log \prod_{i=1}^{n}L(\theta;\mathbf{x}_i,\mathbf{z}_i) ] \\ &= \operatorname{E}_{\mathbf{Z}|\mathbf{X},\mathbf{\theta}^{(t)}} [\sum_{i=1}^n \log L(\theta;\mathbf{x}_i,\mathbf{z}_i) ] \\ &= \sum_{i=1}^n\operatorname{E}_{\mathbf{Z}|\mathbf{X};\mathbf{\theta}^{(t)}} [\log L(\theta;\mathbf{x}_i,\mathbf{z}_i) ] \\ &= \sum_{i=1}^n \sum_{j=1}^2 \overbrace{P(Z_i =j | X_i = \mathbf{x}_i; \theta^{(t)}) \log \underbrace{L(\theta_j;\mathbf{x}_i,\mathbf{z}_i)}_{\text{should be}\\ L(\theta;\mathbf{x}_i,j)}}^\text{see below for discussion of the typo there} \\ &= \sum_{i=1}^n \sum_{j=1}^2 T_{j,i}^{(t)} \big[ \log \tau_j -\tfrac{1}{2} \log |\Sigma_j| -\tfrac{1}{2}(\mathbf{x}_i-\boldsymbol{\mu}_j)^\top\Sigma_j^{-1} (\mathbf{x}_i-\boldsymbol{\mu}_j) -\tfrac{d}{2} \log(2\pi) \big] \end{align}

The conditional probabilities $\mathbb{P}(Z_i=j|X_i,\theta^{(t)})$ appear in the E-step of the EM algorithm because the E-step purpose is to produce the expectation of the complete(d) log-likelihood $\log\,L(\theta;x,z)$ conditionally on the observables $(x_1,\ldots,x_n)$. Hence \begin{align*} \overbrace{\mathbb{E}_{Z|X=x,\theta^{(t)}}}^{\text{i.e., expectation}\\\text{conditional on}\\X=x\ \text{and for the}\\\text{value $\theta^{(t)}$ of $\theta^{(t)}$}}\Big[\log\,\overbrace{L(\theta;x,Z)}^{\text{joint density of}\\ \text{sample of }(X_i,Z_i)}\Big] &= \mathbb{E}_{Z|X=x,\theta^{(t)}}\overbrace{\left[\log\,\prod_{i=1}^n L(\theta;x_i,Z_i)\right]}^{\text{independence of the pairs}\\(X_i,Z_i)\ \text{for}\ i=1,\ldots,n}\\ &= \sum_{i=1}^n \mathbb{E}_{Z|X=x,\theta^{(t)}}\left[\log\, L(\theta;x_i,Z_i)\right]\\ &= \sum_{i=1}^n \mathbb{E}_{Z_i|X_i=x_i,\theta^{(t)}}\left[\log\, L(\theta;x_i,Z_i)\right]\\ &= \sum_{i=1}^n \underbrace{\sum_{j=1}^k \mathbb{P}(Z_i=j|X_i=x_i,\theta^{(t)})\log\, L(\theta;x_i,j)}_{\text{definition of conditional expectation}} \end{align*} Note that there is a typo in the line before last in the Wikipedia derivation of $Q(\theta|\theta^{(t)})$, which should be $$\sum_{i=1}^n \sum_{j=1}^2 \mathbb{P}(Z_i=j|X_i=x_i,\theta^{(t)})\log\, \overbrace{L(\theta;x_i,j)}^{\text{since $Z_i=j$}\\\text{and $\theta$ remains}\\\text{a free variable}}$$rather than$$\sum_{i=1}^n \sum_{j=1}^2 \mathbb{P}(Z_i=j|X_i=x_i,\theta^{(t)})\log\, L(\theta_j;x_i,z_i)$$ Or, with the original notations there, $$\sum_{i=1}^n \sum_{j=1}^2 P(Z_i =j | X_i = \mathbf{x}_i; \theta^{(t)}) \log L(\theta;\mathbf{x}_i,j) \\ $$rather than $$\sum_{i=1}^n \sum_{j=1}^2 P(Z_i =j | X_i = \mathbf{x}_i; \theta^{(t)}) \log L(\theta_j;\mathbf{x}_i,\mathbf{z}_i) \\ $$ As for the question about using the density and not the cdf in the likelihood, this is a generic one. The likelihood of the pairs $(X_i,Z_i)$ is the product of the densities, by definition of the likelihood function. The notation $$P(X_i=x_i|Z_i = 1,\theta^{(t)})$$ is incorrect when $X_i$ is a continuous random variable, but is not to be confused with $$P(X_i\le x_i|Z_i = 1,\theta^{(t)})$$

$\endgroup$
2
  • $\begingroup$ Apologies for being unclear, I have edited the original question to be more precise. $\endgroup$
    – Chris
    May 19, 2018 at 15:48
  • $\begingroup$ Becuse it did not adress my question! You misunderstood what I asked for, and answered the second part of the calculation, not the first part which I asked about. The question was about why is a probabilty equal to a density, not about why they appear in the formula. But your second part now adresses my question, thank you! $\endgroup$
    – Chris
    May 20, 2018 at 12:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.