# Why is it said that maximum likelihood becomes intractable if there are latent variables?

It is said that EM algorithm helps in cases where direct MLE cannot be carried out due to missing/latent variables. However, I could not understand why direct MLE cannot be carried out when there are latent variables.

$$\frac{\partial }{\partial \theta}\prod_{i}^{n}f\left(x_i|\theta\right) \sim \frac{\partial }{\partial \theta} ln \left< \prod_{i}^{n}f\left(x_i|\theta\right) \right> = \frac{\partial }{\partial \theta} \sum_{i}^{n} ln \left< f\left(x_i|\theta\right) \right> =$$ $$\sum_{i}^{n} \frac{\partial }{\partial \theta} ln \left< f\left(x_i|\theta\right) \right> = 0$$

Is it something related to the term $$f\left(x_i|\theta\right>$$?

My Understanding:

Data point $$x_i$$ is actually a collection of variable values. Example: if data point $$x_i$$ is house-metric, variables length, breadth, and height will define it: $$x_i$$=(400,500,200). In case there is a latent variable $$V$$ that affects $$x_i$$, $$V$$ cannot be properly defined in $$x_i$$ since $$V$$ cannot be observed. So, due to that $$x_i$$ will not be complete and $$f\left(x_i|\theta\right>$$ cannot be determined. Hence, MLE becomes intractable.

Is my understanding correct? Or is there a different / additional reason.

• Hint: there is an expectation symbol $\langle \cdot \rangle$ in the expression you copied Commented May 3, 2022 at 8:46
• @J.Delaney Not able to get my head around it even with the hint. Some more hint please? Commented May 3, 2022 at 9:00
• Do you understand what this symbol represents and why it is there ? Commented May 3, 2022 at 9:10
• My understanding is that the product of large number of probability densities can result in very small number (close to 0). So, in order to change product expression into sum, natural logarithm is used since argmax theta with the product expression or the ln result in the same value. Hence, natural logarithm is applied on the product. However, I do not know why <.> is available and its significance. @J.Delaney Commented May 3, 2022 at 9:25
• Your understanding of why $\ln$ is used is correct, although not specifically related to the question. The brackets represent expectation (integration) w.r.t the latent variables - and integrals can't always be calculated in closed form. Commented May 3, 2022 at 9:53

Your understanding is correct that we cannot observe the latent variable, therefore we need to use the EM algorithm. In the E step we estimate our distribution for the latent variable and In the M step we estimate the parameters that will minimize the KL divergence.

To be more clear you can think of $$f\left(x|0\right)$$ in your question, which I believe is the likelihood, as a joint distribution of $$f\left(x,V,0\right)$$. So, your term would be like $$f(x|V,\theta)f(V|\theta)$$. So, $$x$$ is now dependent on $$V$$ which you do not observe.

Hope this helps.

• This can be handled fairly simply with Bayesian models. Commented Feb 29 at 13:18

Your concerned relation between likelihood intractability and the existence of latent variable can be clearly seen with the formulation of variational autoencoders which is approximate Bayesian inference due to likelihood intractability.

one wants to maximize the likelihood of the data $${\displaystyle x}$$ by their chosen parameterized probability distribution $${\displaystyle p_{\theta }(x)=p(x|\theta )}$$... Simple distributions are easy enough to maximize, however distributions where a prior is assumed over the latents $${\displaystyle z}$$ results in intractable integrals. Let us find $${\displaystyle p_{\theta }(x)}$$ via marginalizing over $${\displaystyle z}$$. $${\displaystyle p_{\theta }(x)=\int _{z}p_{\theta }({x,z})\,dz,}$$ where $${\displaystyle p_{\theta }({x,z})}$$ represents the joint distribution under $${\displaystyle p_{\theta }}$$ of the observable data $${\displaystyle x}$$ and its latent representation or encoding $${\displaystyle z}$$.

You see here in addition to your parameter $$\theta$$, your model may have individual local latent variable $$z$$ for every observable data point $$x$$, then often you have intractability mainly due to the integral in a usual high-dimensional latent space of nontrivial applications.