An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.

learn more… | top users | synonyms (1)

5
votes
1answer
107 views

Why is optimizing a mixture of Gaussian directly computationally hard?

Consider the log likelihood of a mixture of guassians: $$l(S_n; \theta) = \sum^n_{t=1}logP(x^{(t)}|\theta) = \sum^n_{t=1}log\sum^k_{i=1}p_iP(x^{(t)}|\mu^{(i)}, \sigma^2_i)$$ I was wondering why it ...
0
votes
0answers
12 views

Maximum likelihood hypothesis vs Expectation maximization

Maximum Likelihood is given by the formula $h_{ML}=arg\space max_{h \in H } \space\space p(D/h)$ I want to transform it in terms of mechanism involved in Expectation maximization. $h_{ML}=arg\space ...
0
votes
0answers
16 views

Possibilities of accelerating EM algortihm

I'm trying to use the EM to estimate some parameters. I've programmed and it delivers. The problem however is that for each run of my programme, it can take either 5 seconds, 1min, 3min or more to ...
1
vote
0answers
48 views

In EM derivation why can I sum over the iid variables in the conditional expectation?

In EM when you take the expectation: $E[\log P(y,x \mid \theta)\mid x, \theta']$ $= \sum\limits_yP(y\mid x, \theta') \log P(y,x\mid \theta)$ I understand this but the following part I don't ...
0
votes
0answers
43 views

Proper likelihood function in acceptance probability of Gibbs Sampler

I have a question about the acceptance ratio used when implementing a random walk M-H in a gibbs sampler to generate sample paths of an unobservable process. When computing the likelihood of a set of ...
0
votes
0answers
18 views

Simulating EM versus listwise deletion--unexpected results

I'm preparing a presentation on missing data strategies and conducted a simulation to compare listwise deletion (LD) to the EM. Here's what I expected based on the literature: Standard errors will ...
1
vote
1answer
28 views

Efficience of Expectation-Maximization algorithm in function of learning dataset size

I have datasets of increasing sizes identically distributed. I have tried to fit a gaussian mixture to these datasets using Expectation-Maximization algorithm. To check the quality of this fit, I ...
2
votes
0answers
20 views

Computing Standard Errors in EM algorithm

I'm applying the EM to a hidden markov chain (the $\mathbf{Z}=\{Z_1,...,Z_n\}$ variable), with observations(the $\mathbf{Y}=\{Y_0,...,Y_n\}$ variable) dependent not only on the hidden markov chain, ...
0
votes
0answers
36 views

Too good results from EM for gaussian mixture

My task is to identify parameters (mean, standard deviation, height) of gaussian peaks in given histogram data with as lowest CV as possible. Number of peaks and approximate means are known (pointed ...
0
votes
0answers
20 views

Learning Conditional Random Fields using EM Algorithm (from unaligned data)

I am trying to learn CRF from the unaligned data in Natural Language Understanding application. There is one paper in this field which does exactly the same, Learning conditional random fields from ...
2
votes
1answer
62 views

Estimating parameters for univariate skew t

How can I solve the MLE for the skew-t distribution via EM? I am comfortable with the EM methods for t, so could someone show it for the skew-t?
1
vote
1answer
77 views

t-distribution parameter estimation

I know there are already several threads on this, but none seem to explicitly cover what I want. I have a set of financial data (pulled straight from Bloomberg) and am trying to fit a t-distribution ...
1
vote
1answer
58 views

Variance of EM mean estimates in a simple mixture of two normals

Consider a mixture of two normal distributions: $ f(x) = p N(x|u_1, S_1) + (1-p) N(x|u_2, S_2) $ where N() is the normal pdf. $p$, $S_2$, and $S_2$ are known. The means are not. You can get the MLE ...
0
votes
1answer
43 views

Expectation maximization and Gaussian mixtures - bad results

I am supposed to find parameters of individual gaussians in a 1D mixture with a known number of components. I use my own implementation of EM algorithm; however, I am not able to find the right ...
0
votes
0answers
22 views

EM Algorithm to maximize the censored likelihood

I want to implement the EM algorithm to maximize the following censored likelihood when the data is normal: The complete data likelihood is: resulting in the expected complete-data log-likelihood ...
9
votes
0answers
52 views

Training a basic Markov Random Field for classifying pixels in an image

I am attempting to learn how to use Markov Random Fields to segment regions in an image. I do not understand some of the parameters in the MRF or why the expectation maximisation I perform fails to ...
0
votes
0answers
20 views

Expectation-Maximization (EM) method for parameter estimation using fuzzy logic

I am sorry if my question is not fit here. If so, please recommend me the correct forum. I am thinking of estimating a fuzzy model using the EM method. I have a set of observations from a nonlinear ...
8
votes
1answer
159 views

Why is Expectation Maximization algorithm guaranteed to converge to minimum, even local?

I have read a couple of explanations of EM algorithm (e.g. from Bishop's Pattern Recognition and Machine Learning and from Roger and Gerolami First Course on Machine Learning). The derivation of EM is ...
1
vote
0answers
34 views

EM on product of multinomials

I have the following conditional density: $$ P(x | \theta, \pi) = \prod_{i=1}^I \prod_{j=1}^J t_{ij}! \prod_{k=1}^K \frac{1}{x_{ijk}!}(\sum_{l=1}^L \theta_{il} \pi_{jkl})^{x_{ijk}} $$ Here, $x$ is ...
0
votes
1answer
26 views

Global search operators for approximate MAP inference?

In complicated Bayesian models, like for instance a hierarchical nonparameteric one, often times it's intractable to do Gibbs or other MCMC sampling methods to convergence. Rather, people tend to do ...
2
votes
0answers
56 views

Confusion about the EM algorithm

I am reading through the EM-Algorithm but on the slide 39, I don't get how $$P(D) = P(D|A)P(A) + P(D|B)P(B)$$ I am trying to understand it in order to get my head around modelling with mixture of ...
0
votes
0answers
62 views

Comparison of Variational Bayes and Expectation Maximization algorithms

I need to learn both the VB and EM methods for Bayesian Networks. Before going into detail of both algorithms, which I am a bit aware of, I need to EXACTLY understand the basic motivations behind ...
10
votes
2answers
170 views

Does MLE always mean we know the underlying PDF of our data, and does EM mean we don't?

I have some simple conceptual questions that I would like clarified regarding MLE (Maximum Likelihood Estimation), and what link it has, if any, to EM (Expectation Maximization). As I understand it, ...
3
votes
1answer
69 views

Understanding Dempster et al. on expectation maximization

I'm reading about expectation maximization from Dempster, Laird and Rubin's original paper which can be found from the following link: http://web.mit.edu/6.435/www/Dempster77.pdf My questions are ...
1
vote
1answer
55 views

What happens when you initialize EM with a consistent estimate?

I have a certain family of models with latent random variables (not observed in the data), which I have a consistent estimator for. I now run EM on top of it, meaning, I get the consistent ...
2
votes
1answer
34 views

Examples of “Hard” Expectation Maximization other than clustering?

Are there examples of learning algorithms(other than k-means clustering) which fit the paradigm of Hard-EM? By hard EM, I mean the variant described in here.
0
votes
0answers
24 views

Sliding window EM algorithm for PLSA?

I'm wondering is there a version of sliding window EM? I'm working on an application involving a stream of text (tweets). We are thinking of using 1 day data as training set, and extract the topics ...
2
votes
2answers
158 views

Clustering with K-Means and EM: how are they related?

I have studied algorithms for clustering data (unsupervised learning): EM, and k-means. I keep reading the following : k-means is a variant of EM, with the assumptions that clusters are ...
0
votes
0answers
45 views

Normal Data with missing values (EM Algorithm)

Suppose we have multivariate normal data (some missing entries, at random), with known covariance matrix. We would like to estimate the mean vector by the EM algorithm. How would one approach this? ...
1
vote
1answer
103 views

EM algorithm with a component for outliers

i have a vector of measurements from one to three classes, which can be modeled by gaussian distributions. There are some outliers in the data. I use the EM algorithm to learn the parameters of the ...
1
vote
0answers
80 views

Simpson's Paradox, Combining data across confounding variable when few values are missing

The statistical analysis of experimental data that I have to perform could be described as follows. Three drug treatments $D_1$, $D_2$ and $D_3$ were tested across three groups $G_1$, $G_2$ and $G_3$. ...
0
votes
0answers
29 views

how to show the following consistency?

It is well-known that maximum likelihood estimate for a mixture model, with the mixture distributions known, and the estimation is done for the mixture coefficients is consistent (I think) -- the ML ...
0
votes
1answer
242 views

How to interpret Hidden Markov Model parameters (transition matrix, emission matrix, and pi values)?

I am working on channel modeling for cognitive radio using HMM. I've written a MATLAB program for forward, backward and Baum-Welch algorithm for multiple sequences. After given some random input and ...
23
votes
4answers
1k views

Numerical example to understand Expectation-Maximization

I am trying to get a good grasp on the EM algorithm, to be able to implement and use it. I spent a full day reading the theory and a paper where EM is used to track an aircraft using the position ...
2
votes
1answer
91 views

How to accomplish unsupervised separation of subpopulations?

I have a dataset drawn from a social network that looks Bimodal on logarithmic scales for all attributes (I'll demonstrate only one here): I know the variable that would give me a clean separation ...
1
vote
2answers
138 views

EM algorithm to estimate discrete Markov chain transition probabilities

I'm surveying algorithms to estimate the transition probabilities of a discrete Markov chain and I found that EM approach could used. However I am not able to find any simple description on how to ...
1
vote
1answer
72 views

How to approximate 0 in transition probability matrix without loss of generality?

In trying to implement Mixture Markov Model, (see question here), I have extreme cases ( e.g. 0's in the Transition Probability Matrix). I have approached this with replacing 0 with 1e-17. However, I ...
2
votes
1answer
109 views

Can log-likelihood function calculated value (M-step) be smaller after 1 EM-iteration?

I am applying a MAP log-likelihood approach in order to fit a Markov mixture model, where objective function to be maximized is given by the formula: $$ L(X|\Theta ...
0
votes
0answers
85 views

Why isn't k-means optimized using gradient descent?

I know k-means is usually optimized using Expectation Maximization. However we could optimize its loss function the same way we optimize any other! I found some papers that actually use stochastic ...
1
vote
0answers
115 views

EM algorithm for mixture of Negative Binomial distributions

I am trying to derive the EM-algorithm of mixtures of negative binomial distribution $Neg\;Bin(r,p)$. I have the updating equations for updating the E-step as well as $p$ and the mixing coefficients ...
3
votes
2answers
165 views

What does it mean exactly, to “build a statistical model” of, say, a series of images?

I would like a meaningful and concise explanation for what it means exactly, when someone says, "We built a statistical model of all our images". I overheard this, (and keep overhearing that ...
1
vote
2answers
114 views

E-step in EM-algorithm using MAP estimate (mixed Markov models), what does it calculate?

I am trying to grasp what exactly is "estimated" in the E-step of the algorithm. According to all definitions, in E-step the "conditional expectation values , or posterior probabilities of the ...
0
votes
1answer
90 views

Implementing EM clustering for a mixture markov model

I have a mixture Markov model (containing K clusters, or components) that I am trying to train, e.g perform clustering over a set of varying length sequences. Each component of the model is a first ...
0
votes
0answers
105 views

Output of Baum-Welch algorithm and clustering of HMM

I have trouble understanding the output of Baum-Welch algorithm in the context of clustering of time series of unequal length using HMM. Suppose I have N sequences with length L_i. An article that ...
1
vote
0answers
79 views

Should I add noise to my truth data before before training my classifier?

My task is to develop a system that will take in a series of measurements and return the probability that an object is a type 1, type 2,... type n. I will refer to the system I have to create as a ...
5
votes
3answers
178 views

Motivation of Expectation Maximization algorithm

In the EM algorithm approach we use Jensen's inequality to arrive at $$\log p(x|\theta) \geq \int \log p(z,x|\theta) p(z|x,\theta^{(k)}) dz - \int \log p(z|x,\theta) p(z|x,\theta^{(k)})dz$$ and ...
1
vote
1answer
92 views

Can first-order Markov chain be considered a special case of a hidden Markov model?

I am trying to apply R depmixS4 package in order to cluster time series with model based clustering. The model consists of K components, each being a first order Markov models. The ...
0
votes
0answers
49 views

Beyond observable and unobservable data, is there any term “semi-observable” defined?

When dealing with data in fields such as Natural Language Processing(NLP) or Speech Recognition(ASR) and trying to model the data using Hidden Markov Model(HMM) one should first make it clear that if ...
1
vote
1answer
104 views

E-step of the stochastic approximation EM

I am reading the paper: Convergence of a stochastic approximation version of the EM algorithm to implement this algorithm for a probability model I already have. In p. 3, the paper summarises the ...
3
votes
2answers
206 views

Why is the expectation maximization algorithm used?

From what little I know the EM algorithm can be used to find the maximum likelihood when setting to zero the partial derivatives with respect to the parameters of the likelihood gives a set of ...