23
votes
Accepted
Relation between MAP, EM, and MLE
Imagine that you have some data $X$ and probabilistic model parametrized by $\theta$, you are interested in learning about $\theta$ given your data. The relation between data, parameter and model is ...
Tim♦
- 138k
23
votes
Accepted
Why typically minimizing a cost instead of maximizing a reward?
Minimising $f(x)$ is entirely equivalent to maximising $-f(x)$, in every aspect: result, numerical precision, computational complexity... everything.
Historically, the convention might have been ...
20
votes
Accepted
Finding category with maximum likelihood method
This is a classic unsupervised learning problem that has a simple maximum likelihood solution. The solution is a motivating example for the expectation maximization algorithm. The process is:
...
19
votes
Accepted
Why Expectation Maximization is important for mixture models?
In principle, both EM and standard optimization approaches can work for fitting mixture distributions. Like EM, convex optimization solvers will converge to a local optimum. But, a variety of ...
18
votes
Numerical example to understand Expectation-Maximization
Here's an example of Expectation Maximisation (EM) used to estimate the mean and standard deviation. The code is in Python, but it should be easy to follow even if you're not familiar with the ...
15
votes
Accepted
Can log likelihood funcion be positive
Simply (just summarizing the comments):
when using probabilities (discrete outcome), the log likelihood is the sum of logs of probabilities all smaller than 1, thus it is always negative
when using ...
15
votes
Finding category with maximum likelihood method
What you are describing is a mixture of two Gaussians.
$$
f(x) = \pi \, \mathcal{N}(\mu_1, \sigma_1^2) + (1 - \pi) \, \mathcal{N}(\mu_2, \sigma_2^2)
$$
where $\pi \in (0, 1)$ is a mixing proportion. ...
Tim♦
- 138k
15
votes
Accepted
Why go through the trouble of expectation maximization and not use gradient descent?
There are several advantages of the EM algorithm over gradient descent:
Monotonic convergence. The EM algorithm never decreases the log-likelihood. This is not necessarily true for gradient descent.
...
14
votes
Accepted
How can I derive the EM algorithm for a mixture of two Bernoulli distributions?
The following derivation was adapted from the references:
"What is the expectation maximization algorithm?" by Chuong B Do & Serafim Batzoglou (2008)
"A Gentle Tutorial of the EM ...
13
votes
Relation between MAP, EM, and MLE
I stumbled across this post, when I tried to give a more comprehensive answer of that topic to a friend of mine. Since the answer written by @Tim♦ is already a bit older and this topic just really ...
13
votes
What is the relationship between VAE and EM algorithm?
What is the relationship between VAE and EM?
$\newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}}
\newcommand{\vx}{\vect{x}}
\newcommand{\vz}{\vect{z}}
\newcommand{\vtheta}{\vect{\theta}}
\newcommand{\...
12
votes
Why isn't k-means optimized using gradient descent?
As the OP mentions, it's possible to solve k-means using gradient descent, and this may be useful in the case of large scale problems.
There are certainly historical reasons for the prevalence of EM ...
11
votes
Accepted
Is this a typo/error in Bishop's book
Indeed, indeed, there is a typo in (9.16) and it should be
$$0=\sum_{n=1}^N \gamma(z_{nk}) {\mathbf \Sigma}_k^{-1} ({\mathbf x}_n-{\mathbf \mu}_k)$$Fortunately, this does not impact the next equation (...
10
votes
Accepted
MCMC in a frequentist setting
As indicated in the many comments, Markov Chain Monte Carlo is a special case of the Monte Carlo method, which is designed to approximate quantities related with a distribution via pseudo-random ...
10
votes
Why do we say EM is a partially non-Bayesian method?
EM is based on a demarginalisation of the (standard or observed) likelihood
$$L^\text{o}(\theta|\mathbf x)=\int_{\mathfrak Z} L^\text{c}(\theta|\mathbf x,\mathbf z)\,\text d\mathbf z \tag{1}$$
...
9
votes
Accepted
EM Algorithm seems to work, but Q is not monotonic. Possible reasons?
It is the case that the incomplete-data log likelihood has to increase at every step, but is not the case that the expected log likelihood has to increase at every step.
The reason why is hidden in ...
8
votes
Accepted
Difference between KDE, MLE and EM for density estimation
KDE is a non-parametric method: it can be used without specifying any assumptions regarding the density your are trying to estimate. To put it bluntly, if you have a set of $N$ observations $\{x_i\}_{...
7
votes
What is the difference between EM and Gradient Ascent?
I wanted to follow up (even though this is some years later) on the OP's second question:
Is there any condition under which they are equivalent?
In fact there is a condition under which they're ...
7
votes
Accepted
EM maximum likelihood estimation for Weibull distribution
I think the answer is yes, if I have understood the question correctly.
Write $z_i = x_i^k$. Then an EM algorithm type of iteration, starting with for example $\hat k = 1$, is
E step: ${\hat z}_i =...
7
votes
Accepted
Why is the expectation step in the EM algorithm called this way?
You are combining both steps. Breaking them out (e.g. see here), you have
E step
$Q(\theta\mid\theta_\text{old})=\sum_Z p(Z\mid X,\theta_\text{old})\log p(X,Z|\theta)$
M step
$\theta_\text{new}=\...
7
votes
Why typically minimizing a cost instead of maximizing a reward?
You tagged this question with the tag "Maximum Likelihood". In maximum likelihood estimation you explicitly maximize an objective function (namely the likelihood).
It just so happens that ...
7
votes
Understanding the details of Expectation Maximization(EM) for estimating the parameters?
The EM method is a generalized algorithm that solves maximum-likelihood problems with latent variables. Suppose that you have a model with some random variables $y$ that you have observed (...
7
votes
Accepted
Understanding the log-likelihood (score) in scikit-learn GMM
(log-) likelihood of a mixture model
You have a model $g_{\theta}$ to describe some data sample $\mathbf{x}$, in this case your mixture model. This model is dependent on it's parameters, in this case ...
6
votes
Motivation of Expectation Maximization algorithm
There is a useful optimisation technique underlying the EM algorithm. However, it's usually expressed in the language of probability theory so it's hard to see that at the core is a method that has ...
6
votes
Is EM feasible when there is no closed form maximization of the expectation of log likelihood?
Answer to Question 1:
The maximization step can be conducted by numerical optimization. In fact, it is possible to incorporate constraints (if any) on the original problem into the numerical ...
6
votes
Accepted
The number of parameters in Gaussian mixture model
Simply do the math.
For each Gaussian you have:
1. A Symmetric full DxD covariance matrix giving (D*D - D)/2 + D parameters (...
6
votes
Why Expectation Maximization is important for mixture models?
I think user20160's answer provides a very good explanation, the most important reason that makes gradient based methods not suitable here is the constraint for covariance matrices to be positive ...
6
votes
Accepted
EM algorithm: Why do we compute the expectation with respect to the conditional distribution?
For your first question: in EM you are alternating between updating $Z$ and updating $\theta$. (Note however that $Z$ is commonly a set of categorical variables, i.e. labels, in EM applications, so it ...
6
votes
Why typically minimizing a cost instead of maximizing a reward?
It's my understanding that the only reason for this distinction is that in numerical analysis, it's the standard to talk about convex optimization rather than concave optimization, even though they ...
6
votes
Accepted
ELBO maximization with SGD
I think you confuse the purpose of the two methods.
Maximizing the ELBO leads to a parameterized class of densities that approximates closely the true distribution, in terms of Kullback-Leibler ...
Only top scored, non community-wiki answers of a minimum length are eligible
Related Tags
expectation-maximization × 602maximum-likelihood × 118
gaussian-mixture-distribution × 90
machine-learning × 72
optimization × 52
mixture-distribution × 46
bayesian × 44
hidden-markov-model × 42
clustering × 41
self-study × 36
likelihood × 36
missing-data × 32
r × 31
latent-variable × 29
probability × 28
mathematical-statistics × 28
normal-distribution × 24
estimation × 23
expected-value × 18
variational-bayes × 18
baum-welch × 17
conditional-expectation × 16
finite-mixture-model × 15
distributions × 14
inference × 14