1
$\begingroup$

From what I read and understood, when we have a discrete hidden variable that we already know its particular value (instead of summing/marginalizing over them) associated with data then it is appropriate to use the Maximum Likelihood Estimation (MLE). For example, in a labeled corpus when let's say Y is the class to be predicted, and it is a hidden variable at the same time for unlabeled new example, but as we have a labeled corpus for training/estimation so we already know the exact value of Y for each example in the dataset/corpus.

Can we say in a labeled corpus (supervised learning), where the discrete hidden variable values are known, we are supposed to use MLE rather than EM?

$\endgroup$
0
$\begingroup$

I think there is a misconception. EM is just an algorithm to do MLE, and we usually use it when a direct MLE is not possible. In this sense it's independent from the problem being supervised or unsupervised.

Let me give this example model: ($i=1,\dots,N$)

\begin{align} c_i &\sim \mathrm{Bernoulli}(p) \\ y_i | c_i &\sim \begin{cases} \mathrm{SymStable}(\alpha, \sigma_0) & c_i = 0 \\ \mathrm{SymStable}(\alpha, \sigma_1) & c_i = 1 \end{cases} \end{align} where SymStable denotes the symmetric centered $\alpha$-stable distribution and $c_i$ denotes the class labels for $y_i$. Our goal is to estimate $\sigma_0$ and $\sigma_1$ when we observe the pairs $(y_i,c_i)$.

While this is clearly a supervised setting ($c_i$ are available), a direct MLE wouldn't work here since the probability density function of the stable distribution is not analytically available. But you can develop an EM algorithm (although approximate) for this model by first rewriting the model in an equivalent way by 'data augmentation'.

I think this paper is an example for such a scenario:

Teimouri, M., Rezakhah, S., & Mohammadpour, A. (2018). Parameter Estimation Using the EM Algorithm for Symmetric Stable Random Variables and Sub-Gaussian Random Vectors. Journal of Statistical Theory and Applications, 17(3), 439-461.

$\endgroup$
  • $\begingroup$ You mentioned "a direct MLE wouldn't work here since the probability density function of the stable distribution is not analytically available." Could you explain in simple words and give more details why if we do not have the probability density function then we can not use MLE but still can use EM (I am not familiar with the SymStable by the way, but I understand that it is a probability distribution with two parameters α,σ)? $\endgroup$ – Mosab Shaheen Apr 4 at 10:07
  • $\begingroup$ What about a discrete hidden variable, is what I asked in my main question applied on a discrete hidden variable? $\endgroup$ – Mosab Shaheen Apr 4 at 10:11
  • $\begingroup$ By the way, in your explanation you wrote ci∼Bernoulli(p) do you mean by ci a value or a variable, because the symbol "~" means "is distributed as" so ci is a variable and hence we have n variable c1,c2,...cn but it seems you are using it mistakenly to represent n values of same variable c ? $\endgroup$ – Mosab Shaheen Apr 4 at 10:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.