23 votes
Accepted

Relation between MAP, EM, and MLE

Imagine that you have some data $X$ and probabilistic model parametrized by $\theta$, you are interested in learning about $\theta$ given your data. The relation between data, parameter and model is ...
Tim's user avatar
  • 138k
23 votes
Accepted

Why typically minimizing a cost instead of maximizing a reward?

Minimising $f(x)$ is entirely equivalent to maximising $-f(x)$, in every aspect: result, numerical precision, computational complexity... everything. Historically, the convention might have been ...
Igor F.'s user avatar
  • 9,069
20 votes
Accepted

Finding category with maximum likelihood method

This is a classic unsupervised learning problem that has a simple maximum likelihood solution. The solution is a motivating example for the expectation maximization algorithm. The process is: ...
AdamO's user avatar
  • 62.5k
19 votes
Accepted

Why Expectation Maximization is important for mixture models?

In principle, both EM and standard optimization approaches can work for fitting mixture distributions. Like EM, convex optimization solvers will converge to a local optimum. But, a variety of ...
user20160's user avatar
  • 32.4k
18 votes

Numerical example to understand Expectation-Maximization

Here's an example of Expectation Maximisation (EM) used to estimate the mean and standard deviation. The code is in Python, but it should be easy to follow even if you're not familiar with the ...
Alex Riley's user avatar
15 votes
Accepted

Can log likelihood funcion be positive

Simply (just summarizing the comments): when using probabilities (discrete outcome), the log likelihood is the sum of logs of probabilities all smaller than 1, thus it is always negative when using ...
Benoit Sanchez's user avatar
15 votes

Finding category with maximum likelihood method

What you are describing is a mixture of two Gaussians. $$ f(x) = \pi \, \mathcal{N}(\mu_1, \sigma_1^2) + (1 - \pi) \, \mathcal{N}(\mu_2, \sigma_2^2) $$ where $\pi \in (0, 1)$ is a mixing proportion. ...
Tim's user avatar
  • 138k
15 votes
Accepted

Why go through the trouble of expectation maximization and not use gradient descent?

There are several advantages of the EM algorithm over gradient descent: Monotonic convergence. The EM algorithm never decreases the log-likelihood. This is not necessarily true for gradient descent. ...
Cliff AB's user avatar
  • 20.9k
14 votes
Accepted

How can I derive the EM algorithm for a mixture of two Bernoulli distributions?

The following derivation was adapted from the references: "What is the expectation maximization algorithm?" by Chuong B Do & Serafim Batzoglou (2008) "A Gentle Tutorial of the EM ...
mhdadk's user avatar
  • 4,885
13 votes

Relation between MAP, EM, and MLE

I stumbled across this post, when I tried to give a more comprehensive answer of that topic to a friend of mine. Since the answer written by @Tim♦ is already a bit older and this topic just really ...
Yannik Suhre's user avatar
13 votes

What is the relationship between VAE and EM algorithm?

What is the relationship between VAE and EM? $\newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\vx}{\vect{x}} \newcommand{\vz}{\vect{z}} \newcommand{\vtheta}{\vect{\theta}} \newcommand{\...
Euler_Salter's user avatar
  • 2,196
12 votes

Why isn't k-means optimized using gradient descent?

As the OP mentions, it's possible to solve k-means using gradient descent, and this may be useful in the case of large scale problems. There are certainly historical reasons for the prevalence of EM ...
user20160's user avatar
  • 32.4k
11 votes
Accepted

Is this a typo/error in Bishop's book

Indeed, indeed, there is a typo in (9.16) and it should be $$0=\sum_{n=1}^N \gamma(z_{nk}) {\mathbf \Sigma}_k^{-1} ({\mathbf x}_n-{\mathbf \mu}_k)$$Fortunately, this does not impact the next equation (...
Xi'an's user avatar
  • 105k
10 votes
Accepted

MCMC in a frequentist setting

As indicated in the many comments, Markov Chain Monte Carlo is a special case of the Monte Carlo method, which is designed to approximate quantities related with a distribution via pseudo-random ...
Xi'an's user avatar
  • 105k
10 votes

Why do we say EM is a partially non-Bayesian method?

EM is based on a demarginalisation of the (standard or observed) likelihood $$L^\text{o}(\theta|\mathbf x)=\int_{\mathfrak Z} L^\text{c}(\theta|\mathbf x,\mathbf z)\,\text d\mathbf z \tag{1}$$ ...
Xi'an's user avatar
  • 105k
9 votes
Accepted

EM Algorithm seems to work, but Q is not monotonic. Possible reasons?

It is the case that the incomplete-data log likelihood has to increase at every step, but is not the case that the expected log likelihood has to increase at every step. The reason why is hidden in ...
jbowman's user avatar
  • 38.6k
8 votes
Accepted

Difference between KDE, MLE and EM for density estimation

KDE is a non-parametric method: it can be used without specifying any assumptions regarding the density your are trying to estimate. To put it bluntly, if you have a set of $N$ observations $\{x_i\}_{...
Camille Gontier's user avatar
7 votes

What is the difference between EM and Gradient Ascent?

I wanted to follow up (even though this is some years later) on the OP's second question: Is there any condition under which they are equivalent? In fact there is a condition under which they're ...
Lucas Roberts's user avatar
7 votes
Accepted

EM maximum likelihood estimation for Weibull distribution

I think the answer is yes, if I have understood the question correctly. Write $z_i = x_i^k$. Then an EM algorithm type of iteration, starting with for example $\hat k = 1$, is E step: ${\hat z}_i =...
DavidF's user avatar
  • 86
7 votes
Accepted

Why is the expectation step in the EM algorithm called this way?

You are combining both steps. Breaking them out (e.g. see here), you have E step $Q(\theta\mid\theta_\text{old})=\sum_Z p(Z\mid X,\theta_\text{old})\log p(X,Z|\theta)$ M step $\theta_\text{new}=\...
GeoMatt22's user avatar
  • 13k
7 votes

Why typically minimizing a cost instead of maximizing a reward?

You tagged this question with the tag "Maximum Likelihood". In maximum likelihood estimation you explicitly maximize an objective function (namely the likelihood). It just so happens that ...
Steve Cox's user avatar
  • 371
7 votes

Understanding the details of Expectation Maximization(EM) for estimating the parameters?

The EM method is a generalized algorithm that solves maximum-likelihood problems with latent variables. Suppose that you have a model with some random variables $y$ that you have observed (...
Riccardo Sven Risuleo's user avatar
7 votes
Accepted

Understanding the log-likelihood (score) in scikit-learn GMM

(log-) likelihood of a mixture model You have a model $g_{\theta}$ to describe some data sample $\mathbf{x}$, in this case your mixture model. This model is dependent on it's parameters, in this case ...
deemel's user avatar
  • 2,704
6 votes

Motivation of Expectation Maximization algorithm

There is a useful optimisation technique underlying the EM algorithm. However, it's usually expressed in the language of probability theory so it's hard to see that at the core is a method that has ...
Dan Piponi's user avatar
6 votes

Is EM feasible when there is no closed form maximization of the expectation of log likelihood?

Answer to Question 1: The maximization step can be conducted by numerical optimization. In fact, it is possible to incorporate constraints (if any) on the original problem into the numerical ...
Mark L. Stone's user avatar
6 votes
Accepted

The number of parameters in Gaussian mixture model

Simply do the math. For each Gaussian you have: 1. A Symmetric full DxD covariance matrix giving (D*D - D)/2 + D parameters (...
nyro_0's user avatar
  • 176
6 votes

Why Expectation Maximization is important for mixture models?

I think user20160's answer provides a very good explanation, the most important reason that makes gradient based methods not suitable here is the constraint for covariance matrices to be positive ...
dontloo's user avatar
  • 16.3k
6 votes
Accepted

EM algorithm: Why do we compute the expectation with respect to the conditional distribution?

For your first question: in EM you are alternating between updating $Z$ and updating $\theta$. (Note however that $Z$ is commonly a set of categorical variables, i.e. labels, in EM applications, so it ...
GeoMatt22's user avatar
  • 13k
6 votes

Why typically minimizing a cost instead of maximizing a reward?

It's my understanding that the only reason for this distinction is that in numerical analysis, it's the standard to talk about convex optimization rather than concave optimization, even though they ...
Cliff AB's user avatar
  • 20.9k
6 votes
Accepted

ELBO maximization with SGD

I think you confuse the purpose of the two methods. Maximizing the ELBO leads to a parameterized class of densities that approximates closely the true distribution, in terms of Kullback-Leibler ...
Forgottenscience's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible