An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.
1
vote
1answer
11 views
In the expectation step, why do we sometimes assign the data to a component (i.e. complete the data) instead of calculating the expected value?
Let $Y|X$ be a mixture distribution conditional on covariates $X$, with distribution function $Y(x; \sigma, \psi, \phi) = \alpha Y_1(x; \psi) + (1 - \alpha) Y_2(x; \phi)$, for the averaging parameter $...
0
votes
0answers
9 views
How to use the estimate given by the EM algorithm to guess at the missing value
Text: Computational Statistics 2E by Givens and Hoeting
Example 4.1: Simple Exponential Density
The set-up is as follows: Suppose that $Y_1, Y_2 \overset{\text{iid}}{\sim} \rm{Exp}(\theta)$ and that ...
2
votes
1answer
51 views
Gaussian Mixture Model
with the following code I fit a Gaussian Mixture Model to arbitrarily created data. The code is working. The only thing I encounter is that during the calculation of the multivariate_normal I ...
2
votes
0answers
14 views
Multivariate mixture models
I am new to mixture modeling and have successfully used bernoulli mixture models to cluster datasets of binary data.
My real purpose, though is to cluster datasets with mixed data types: normal, ...
0
votes
0answers
24 views
Sample covariance matrix notation
I do not understand this notation for the sample covariance matrix (from Artificial Intelligence: A Modern Approach, Peter Norvig and Stuart J. Russell, Section 20.3, EM algorithm):
$\Sigma_{i} = \...
0
votes
0answers
12 views
Likelihood scaling for Bernoulli Mixture Model to avoid underflow
I am very new to Expectation Maximization and struggling with how to scale the likelihood calculations to avoid numeric underflow.
I am trying to create a Bernoulli Mixture Model for sparse, binary ...
0
votes
0answers
11 views
Confusion over expectation maximisation algorithm
first of all, apologies if this is the wrong place for this.
I've been reading around my actuarial studies, and came across the expectation maximisation algorithm. I first read this
article, which ...
2
votes
1answer
71 views
Why is the prior omitted from this Bayes rule?
I'm trying to understand the EM algorithm. I've found a tutorial on it. It goes like this:
Two coins (A & B). 5 rounds of flipping 10 times. We forgot, however, which coin was flipped each round. ...
3
votes
0answers
21 views
Termination Condition(s) for Expectation Maximization
What are good criteria for deciding when to terminate the expectation-maximization algorithm.
I know that the idea is that you should terminate when the change in the data log likelihood is "small" ...
0
votes
1answer
38 views
Usefulness of EM algorithm
I wonder how EM make things easier when we are finding the MLE with missing data.
Let $Z$ be the complete data, $Y = Y(Z)$ the observed data, and $\theta$ the parameter to be estimated.
For the MLE, ...
4
votes
1answer
467 views
Is this a typo/error in Bishop's book
I am currently going through the chapter 9 - Mixture Models and EM from Bishop's book - Pattern Recognition and Machine Learning (2006).
I could not understand the maximization step with respect to ...
1
vote
1answer
36 views
Distribution function or density in Mixed Distribution EM
In this calculation
https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm#E_step
a probability, $P(Z_i = j|X_i=x_i;\theta^{(t)})$ is evaluated using bayes theorem, and then each ...
2
votes
0answers
57 views
Using Naive Bayes classifier for unsupervised learning
I was going through this article to learn about how the EM algorithm can be used to use the Naive Bayes algorithm for unsupervised learning. Suppose we have the following data without labels:
1 0 1 1
...
2
votes
1answer
53 views
What are some applications of unsupervised HMMs?
Supervised HMMs can be applied to many problems like POS tagging and OCR (optical character recognition).
I've learned that HMMs can be trained unsupervisedly using EM (Baum-Welch algorithm), what ...
1
vote
1answer
283 views
Estimating truncation point in Gaussian mixture
I have data modeled as a mixture of two Gaussian distributions. The data is "clipped" i.e., there is data only for values greater than a threshold $t$, even though it is feasible for data to exist in ...
0
votes
2answers
34 views
Why different initial parameters of Expectation–Maximization (EM) result in different clusters? [closed]
I'm having a hard time understanding conceptually why we get different clusters when we start the algorithm with different initial parameters. Can anyone explain the mechanisms behind it to me a ...
1
vote
2answers
106 views
Why the EM algorithm instead of a more direct computation? [duplicate]
Assume we have a probability distribution $P(y,z|\theta)$,
where $x$ is the total set of variables divided into observable variables $y$
and hidden variables $z$,
and data on observable variables $y$.
...
1
vote
1answer
56 views
Covariance matrix of image data is not positive definite matrix
I've really hit the wall here and need help with direction :).
I am trying to use mvnpdf as part of basic EM algorithm but the covariance matrix of data seems to be not positive definite. There are ...
1
vote
1answer
91 views
EM Algorithm seems to work, but Q is not monotonic. Possible reasons?
I have implemented Expectation maximization to fit some of the parameters of a linear Gaussian state space model using Kalman filtering / smoothing.
The model is:
$x(t) = Ax(t - 1) + w(t); w(t) \sim ...
1
vote
0answers
3 views
How to estimate the observation level data generating when there is unequal sampling of data for each record
I am looking at bid amounts for cars sold at a sealed auction (only the seller has all the information). I am trying to predict the number and magnitude of offers that a car will receive. Some of the ...
0
votes
0answers
18 views
EM algorithm coupled latent variables
Assume we have a set of latent variables $X = \{x_1,x_2,\dots,x_n\}$, a set of model parameters $\theta = \{\theta_1,\theta_2,\dots,\theta_m\}$ and a set of observed variables $Y= \{y_1,y_2,\dots,y_n\}...
4
votes
1answer
108 views
How to picture EM algorithm and KL-divergence geometrically?
In reading up on the Expectation-Mmaximization algorithm on Wikipedia, I read this short and intriguing line, under the subheading "Geometric Intuition":
In information geometry, the E step and the ...
0
votes
0answers
37 views
Estimate parameters from truncated normal sample [duplicate]
I have a question like this, $X \sim N(\mu,\sigma^2)$ with unknown parameters. Now, a sample of size $m$ generated from X, but filter by X < T, i.e., any number larger than T will be ignored and ...
1
vote
0answers
114 views
GMM EM algorithm complexity per iteration
I was fitting GMM clusters with diagonal covariance on my data using EM with $n$ (=5e6) points, each having $m$ (=160) ...
1
vote
1answer
103 views
EM algorithm with constraints
Lets say that we want to constraint some of the variables in a model, we denote the model's parameters by $\theta \in \mathbb{R}^n$ and we want to train the model on input data $X$.
Normally we would ...
0
votes
0answers
19 views
Conditional expectations of the missing values in EM
I'm trying to figure out the EM algorithm for the right-censored linear regression. In order to perform the M step I need to find the conditional expectations of the missing values:
My question is ...
0
votes
2answers
64 views
latent dirichlet allocation: complexity and implementation details
I was confused by how LDA (by the original variational inference) can be implemented in a way such that the number of operations for each document $j$ is $\mathcal{O}(N_j~K)$, where $N_j$ is the ...
0
votes
0answers
20 views
Notation of “integrate over the unobserved data” in Expectation-Maximization
In the context of Expectation-Maximization, I have always written:
$$
p(y |\phi) = \int_z p(y, z | \phi) dz
$$
Reading through some theoretical papers, though, I find this notation
\begin{align}
g(y |...
2
votes
2answers
107 views
This is Gaussian mixture model?
Here is a problem that I am looking at.
Is this model really commonly known as a Gaussian mixture model (the one often appears as an illustration of EM algorithm)?
I am confused because Gaussian ...
1
vote
2answers
90 views
Variational inference: how to rewrite ELBO?
I am reading this paper on variational inference and this website.
One thing I am confused about is how they get to decompose ELBO, where $ELBO(q) = E_q[log~p(z,x)] - E_q[log~q(z)]$, when focusing ...
4
votes
1answer
90 views
Does marginalization of some of the latent variables improve convergence in EM?
Given a likelihood to maximize
$$
\log p(x | \theta)
$$
Imagine that, in order to apply EM, we can augment the model with one or two latent variables. In that case, we can derive two lower bounds:
$...
0
votes
0answers
18 views
question regarding the expected value of sufficient statistics given complete and incomplete data in EM
I was reading the EM algorithm.
I have a question that I couldn't figure out.
Say given the exponential family $f(x \vert \theta)$ with
$log f(x \vert \theta) = (S(x)) ^T \theta - a(\theta) +b(x) ...
1
vote
0answers
62 views
Expectation Maximization for Urn Problem
I am currently investigating the following mental exercise:
An urn is filled with N balls. Each ball possesses a number and it is either red or green.
There are M color detectors. Each detector ...
1
vote
1answer
104 views
Why use EM algorithm instead of just plain old ML for mixture model?
Let's say I have some [multivariate] data and want to fit a GMM to it. So I have $P_x=\sum_{i=1}^{n}\alpha_i{N(x;\theta_i)}$, where $x$ is an observation from the data, $\theta_i$ is the mean and ...
0
votes
1answer
57 views
EM algorithm: when M-step is difficult
I'm new to EM algorithm and I'm wondering if there is an easy way for the M-step if the likelihood function is complicated (especially when closed form solution is not easy to find). It seems that ...
0
votes
1answer
31 views
How I can find t to maximize this Q-function?
I am wondering Q is a Q-function, and
$Y = Q(\frac{t-1}{0.0894})(1-Q(\frac{t}{0.0894}))$
How can I find $t$ such that it maximizes $Y$?
0
votes
0answers
18 views
RVM-regression: Approximation of hyperparameters using the EM vs direct differentiation approach
I have implemented RVM for regression following this patent paper from Tipping.
I used the datasets Tipping also used in this paper to compare the EM and the direct differentiation update rules.
The ...
0
votes
1answer
57 views
How to prevent the creation of redundant mixtures while training a GMM?
I'm currently trying to train a GMM(UBM) with 1024 Gaussian mixtures for speaker verification.
However, after training the GMM, it appears that some mixtures are useless/redundant.
(little to no ...
0
votes
0answers
35 views
Basic question regarding certain EM execution
I was given the following question on the topic of EM:
Denote by $X_1$, $X_2$ two random variables. Assumed you have $N_1$ I.I.D.
samples of $X_1$, and $N_2$ samples of $X_2$. In these, you observed $...
0
votes
1answer
32 views
Avoiding burn-ins in MC-EM
In Monte Carlo - EM, we use a Monte Carlo sampler in the E-step to approximate the posterior distribution of the latent variables.
The algorithm goes iterates through
E-step: $Z_1,...Z_m \sim p(Z | ...
6
votes
1answer
184 views
Fitting Gaussian mixture models with dirac delta functions
I was told that using gradient methods for Gaussian mixture models may end up with Dirac delta function(s). I hadn't thought of this problem before, but when I verify this, it does seem to be a ...
2
votes
1answer
110 views
EM Algorithm - Derivative wrt covariance
I am struggling trying to find the derivative of the expression below $\frac{\partial}{\partial \sum_k^{-1}} $ wrt the covariance matrix $\sum_k^{-1}$
$ \max \sum_{n=1}^N \sum_{k=1}^K q_{kn} \log ( \...
0
votes
1answer
95 views
Can the plot of the log-likelihood told something about the good starting values in EM
As known, EM algorithm is sensitive to the starting values. One method to select the starting values is to run EM several times using different starting values each time. Then, the select the one that ...
1
vote
1answer
79 views
Can log likelihood funcion be positive [duplicate]
I have a mixture data. I used EM to estimate the model parameters. When I calculate the log likelihood function, I found that the values is positive. So, is that ok. Can the log likelihood function ...
1
vote
0answers
19 views
Can the estimation of parameters in EM be decreases and increases during the estimation process
I am working on EM algorithm (manually implemented in R). I saw that the estimation of the parameters vary from iteration to another. That is, it decreases at some iteration and then start increasing ...
0
votes
1answer
33 views
EM clustering, initial cluster position
Does there exist an implementation (EM-clustering) that allows you to specify the initial cluster positions beforehand? The datasets relevant for me are two dimensional and since just assign random ...
1
vote
0answers
77 views
loglikelihood decrease very slightly in EM algorithm
I am working with a very large and complicated function. I am using EM algorithm to estimate the model parameters. The EM works very well. However, after 27 iteration I see that the values of the ...
0
votes
0answers
112 views
Hidden Markov Models (HMM) with exponential family distributions
HMM learning with Baum Welch (EM) is well known, but the parameterization is all the elements in the transition and emission matrices $a_{ij}, b_i(k)$, that is the number of parameters is $\mathcal{O}(...
2
votes
1answer
108 views
Expectation maximization when support of likelihood is parameterized
I'm trying to solve a homework problem and I'm not getting the answer I expect.
The problem is from Pattern Classification by Duda,Hart,Stork (problem 3.47)
Consider $\mathcal{D} = \left\{\begin{...
2
votes
1answer
213 views
EM for mixture of negative binomial distributions in R
I'm having problems with a fairly basic EM algorithm for a mixture of negative binomial distributions. Given the mean-dispersion parametrisation of the negative binomial distribution, we have a model ...
