I have some simple conceptual questions that I would like clarified regarding MLE (Maximum Likelihood Estimation), and what link it has, if any, to EM (Expectation Maximization).

As I understand it, if someone says "We used the MLE", does that automatically mean that they have an explicit model of their data's PDF? It seems to me that the answer to this is yes. Put another way, if at any time someone says "MLE", it is fair to ask them what PDF they are assuming. Would this be correct?

Lastly, on EM, my understanding is that in EM, we do not actually know - or need to know, the underlying PDF of our data. This is my understanding.

Thank you.

  • 1
    $\begingroup$ The "M" in EM stands for Maximization ... of likelihood. To write down a likelihood we need a pdf. EM is a way of finding MLEs in the presence of 'unobservables' in some sense (which are filled in in the E-step). That is, to use EM you need an explicit model. $\endgroup$ – Glen_b -Reinstate Monica Dec 17 '13 at 17:31
  • $\begingroup$ @Glen_b Thanks Gleb_b. So, 1) would it be correct to say, that in EM, as in MLE, we always assume some model of the data's PDF"? Meaning that if someone says "We used MLE/EM", we can fairly ask, "What PDFs did you assume". Would this be a correct assessment? 2) Lastly, in regards to EM, I believe the unobservables you are referring to are the probabilities of particular PDFs making up the mixture, correct? Thanks in advance. $\endgroup$ – Creatron Dec 17 '13 at 18:20
  • $\begingroup$ Note that there are non-parametric maximum likelihood methods. Look up Kaplan-Meier. $\endgroup$ – soakley Dec 17 '13 at 22:01
  • $\begingroup$ Creatron - on (1) Note that EM is an algorithm for computing MLEs that would otherwise be difficult to deal with. In either case, I'd ask the slightly more general question 'what was your model?', since it's quite possible the model to be more complex than some single pdf. On (2) The EM algorithm doesn't only apply to mixtures; it's more general than that. $\endgroup$ – Glen_b -Reinstate Monica Dec 18 '13 at 0:48

The MLE method can be applied in cases where someone knows the basic functional form of the pdf (e.g., it's Gaussian, or log-normal, or exponential, or whatever), but not the underlying parameters; e.g., they don't know the values of $\mu$ and $\sigma$ in the pdf: $$f(x|\mu, \sigma) = \frac{1}{\sqrt{2\pi\sigma^{2}}} \exp\left[\frac{-(x-\mu)^{2}}{2 \sigma^{2}}\right]$$ or whatever other type of pdf they are assuming. The job of the MLE method is to choose the best (i.e., most plausible) values for the unknown parameters, given the particular data measurements $x_{1}, x_{2}, x_{3}, ...$ which were actually observed. So to answer your first question, yes, you are always within your rights to ask someone what form of pdf they are assuming for their maximum likelihood estimate; indeed, the estimated parameter values that they tell you aren't even meaningful unless they first communicate that context.

The EM algorithm, as I've seen it applied in the past, is more of a sort of meta algorithm, where some of the metadata is missing, and you have to estimate that also. So, for example, perhaps I have a pdf which is a mixture of several Gaussians, e.g.: $$ f(x|A_{1},...,A_{N},\mu_{1},...,\mu_{N}, \sigma_{1},...\sigma_{N}) = \sum_{k=1}^{N} \frac{A_{k}}{\sqrt{2\pi\sigma_{k}^{2}}} \exp\left[\frac{-(x-\mu_{k})^{2}}{2 \sigma_{k}^{2}}\right] $$ Superficially, except for the addition of the amplitude parameter $A_{k}$, this looks a lot like the previous problem, but what if I told you that we also don't even know the value of $N$ (i.e, the number of modes in the Gaussian mixture) and we want to estimate that from the data measurements $x_{1}, x_{2}, x_{3}, ...$ too?

In this case, you have a problem, because each possible value of $N$ (this is the "meta" part that I was alluding to above) really generates a different model, in some sense. If $N=1$, then you have a model with three parameters ($A_{1}$, $\mu_{1}$, $\sigma_{1}$) whereas if $N=2$, then you have a model with six parameters ($A_{1}$, $A_{2}$, $\mu_{1}$, $\mu_{2}$, $\sigma_{1}$, $\sigma_{2}$). The best fit values that you obtain for ($A_{1}$, $\mu_{1}$, $\sigma_{1}$) in the $N=1$ model can't directly be compared to the best fit values that you obtain for those same parameters in the $N=2$ model, because they are different models with a different number of degrees of freedom.

The role of the EM algorithm is to provide a mechanism for making those types of comparisons (usually by imposing a "complexity penalty" that prefers smaller values of $N$) so that we can choose the best overall value for $N$.

So, to answer your original question, the EM algorithm requires a less precise specification of the form of the pdf; one might say that it considers a range of alternative options (e.g., the option where $N=1$, $N=2$, $N=3$, etc.) but it still requires you to specify something about the basic mathematical form of those options--you still have to specify a "family" of possible pdfs, in some sense, even though you are letting the algorithm decide for you which "member" of the family provides the best fit to the data.

  • $\begingroup$ Some follow ups on your Excellent answer @stachyra: (1): The second equation (with the summation) - Is this the PDF of your mixture? (Meaning that $\sum A_k = 1$?) (2): In regards to the EM algorithm mentioned here - a little confused - is the value $N$ given as an input to EM in the beginning, or is this something that EM will spit out as an output in the end? $\endgroup$ – Creatron Dec 17 '13 at 18:49
  • $\begingroup$ (3) Again for EM, when you say "specify the family of possible PDFs" for the EM, does this mean that we give it "possibilities' to work with, for example, "This data is made of two gaussians and one poisson", or "This data is made of 3 chi-squared PDFs and 1 gaussian", etc? This is confusing because it means we specify $N$, which I take it from your post is something EM gives us... $\endgroup$ – Creatron Dec 17 '13 at 18:49
  • $\begingroup$ 1) Yes, this is the pdf of my mixture, and yes, $\sum A_{k} = 1$. 2) In practice, you usually give a min/max value of $N$ for the algorithm to consider, and it loops through all allowed values to find the best one. 3) In most cases, the different possibilities that you are trying to choose between are just the different possible values of $N$; e.g., "$N=4$ gives a good fit, but $N=5$ is even better". If you want to consider alternatives that include a more heterogeneous collection of functional forms, in principle that's possible too, but trickier to implement in practice. $\endgroup$ – stachyra Dec 17 '13 at 19:13
  • $\begingroup$ Thank you stachyra. Last question, the PDF of out data mixture (given in your second equation made up of a weighted sum of PDFs), is NOT the same as the joint PDF of all the samples of our data, which is a product of their PDFs, correct? (Assume the data samples are IID). $\endgroup$ – Creatron Dec 17 '13 at 19:30
  • $\begingroup$ No, not at all--they're two completely different things. The joint pdf that you are describing sounds much more similar to the form of the likelihood function used in MLE. A textbook might be helpful to you here. For MLE, I like chapter 10 of "Data Reduction and Error Analysis for the Physical Sciences" by Philip R. Bevington and D. Keith Robinson, or section 6.1 of "Statistical Data Analysis" by Glen Cowan. For a specific example of how to do one particular type of EM implementation, I like this explanation, sections 2 through 5. $\endgroup$ – stachyra Dec 17 '13 at 20:03

MLE requires knowledge of at least the marginal distributions. When using MLE, we usually estimate the parameters of a joint distribution by making an iid assumption, then factoring the joint distribution as a product of the marginals, which we know. There are variations, but this is the idea in most cases. So MLE is a parametric method.

The EM algorithm is a method for maximizing the likelihood functions that come up as part of a MLE algorithm. It is often (usually?) used for numerical solutions.

Whenever we use MLE, we need at least the marginal distributions, and some assumption about how the joint is related to the marginals (independence, etc.). Therefore both methods rely on knowledge of distributions.

  • $\begingroup$ Thanks @Charles that makes sense. What does it mean then when people talk about "non-parametric MLE". That phrase doesn't makes sense at first glance. MLE always estimate a parameter of the distribution, right? $\endgroup$ – Creatron Dec 17 '13 at 22:32
  • $\begingroup$ They may be talking about ELE (Empirical Likelihood Estimation). I've never used it; I'll try to explain if necessary. Otherwise I'm not sure. $\endgroup$ – Charles Pehlivanian Dec 18 '13 at 2:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.