An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.

learn more… | top users | synonyms (2)

0
votes
1answer
7 views

EM algorithm for maximizing the likelihood of Multivariate Hawkes process

I am trying to model data with multivariate Hawkes distribution. Take the below example. I am able to compute likelihood but dont know how to maximize it. ...
0
votes
0answers
13 views

QDA vs EM with Gaussian likelihoods

QDA (quadratic discriminant analysis) assumes that the K different classes are generated by K different multivariate Gaussians, each with potentially different mean vector and covariance matrix. If ...
0
votes
0answers
8 views

How to compute (only) the transition matrix of an HMM in a supervised manner?

I am currently working with HMM (Hidden Markov Chain especially) to perform image classification. So far, I have used the Viterbi algorithm to do this as I know the parameters of all class-...
0
votes
1answer
27 views

Explanation of the EM algorithm

I've seen this paper's straightforward description of the EM algorithm cited countless times now to explain EM (figure below). But it's only causing me more confusion because I have trouble seeing how ...
0
votes
1answer
25 views

Is EM feasible when there is no closed form maximization of the expectation of log likelihood?

In every example I've seen of expectation maximization, the E step concludes with an expression of the expectation of log likelihood ( $Q(\theta | \theta^{(t)})$ ) for which a maximum w.r.t. $\theta$ ...
0
votes
1answer
37 views

Does constrained EM algorithm work with bad initial inputs?

When trying to perform constrained optimization using EM algorithm, does EM work if the initial solution (x0) violates the constraints?
1
vote
0answers
29 views

Finding a Scalable Approach to a modified coin-flip problem using MLE model

and thank you in advance for your help! I am interested in applying MLE to estimate parameters in a "modified" coin flip model, but have been having difficulties scaling the solution. The problem ...
1
vote
0answers
13 views

mixed noise and gaussian

I have a large number of data sets. Each data set has something 200K data points lying in a square times a circle. The square is solid $I\times I$. The circle $S^1$ is hollow (dim 1). By reasoning ...
0
votes
0answers
17 views

Learn a HMM with fixed emission probabilities constraint

Suppose we want to learn a HMM with the emission matrix is fixed. Can I use the Baum-Welch to estimate the transition probabilities $a_{ij}$ by skipping the $b_{ik}$ values update step at each ...
3
votes
0answers
36 views

Baum-Welch algorithm variation for Hidden Markov model with reward

Following my previous question on the subject I would like to get your feedback on the following alternative solution. (The original solution to this question is the usage of the POMDP model proposed ...
1
vote
2answers
49 views

Clustering groups of observations

I am having a situation where my data points consist of $r$ groups, that we want to force the observations within a group to be in the same cluster, with $n_r$ observations in each group. So the idea ...
2
votes
0answers
9 views

Expectation Maximization Gaussian Mixture Example

I am a biologist trying to understand expectation maximization for a mixture of two Gaussian distributions. I think I understand how to deal with the means of the two distributions, but I don't know ...
0
votes
0answers
32 views

Using the logLikelihood in the EM algorithm

I was obtaining parameter estimates via a EM like algorithm, but taking the expectation of the posterior distribution rather than the logPosterior distribution. I know that this now does not guarantee ...
1
vote
0answers
14 views

Can expectation maximization be used to optimize a quadratic function? [duplicate]

My knowledge about Expectation Maximization (EM) is limited, from my understanding, EM is just an algorithm to do optimization. It works well when we have some hidden / latent variables, such as ...
0
votes
0answers
15 views

How to identify parameters of distributions of a mixture when many samples are sum of the samples?

I have a list of values which are samples from 4 different normal distributions. The list also contains sum of two or three sample values. I would like to identify parameters of the distributions. An ...
3
votes
1answer
53 views

Advantages and disadvantages of EM algorithm vs trust region methods for nonlinear optimization

I have a set of observations X that I believe were generated by a mixture of several probability distributions (specifically, two von mises and one uniform). I'd like to find the maximum likelihood ...
0
votes
0answers
15 views

Soft Expectation Maximization + final hard assignments = Hard Expectation Maximization?

I am studying a given model $\mathcal{M}$ where the authors infer some parameters $\boldsymbol{\theta}$ that are common to all individuals in a population. The parameters are estimated by MLE, but ...
0
votes
2answers
27 views

Expectation Maximization-Log Likehood interpretation

I am using EM algorithm in weka for genomic data, get the results in the images, but a don't know how interpret the log likehood index. Is better when is higher or lower, negative or positive. How ...
1
vote
0answers
47 views

EM versus other methods of optimization

What are some good examples of likelihoods which are easily maximized by EM but not by other methods of optimization (e.g., gradient ascent) and vice versa?
3
votes
0answers
29 views

Indirect solution for maximum entropy through sampling?

Is there a way to sample from a finite set $\{A,B,C,D\}$ such that the limiting empirical proportions converges to the maximum entropy solution of their probabilities consistent with known constraints?...
0
votes
0answers
20 views

EM algorithm help - Plot of expectation

Can someone plot an expectation of a function and show me how maximizing it = maximizing the lower bound of its likelihood in the EM algorithm ? I don't know how to plot the expectation of a function ...
0
votes
0answers
20 views

EM Algorithm - E step

In the E-step, why are there 2 different thetas ? Isnt the expectation of a function $ E[x] = \int xp(x) ?$ If I know x then I would know p(x) as well right ? Based on what I am reading $ E[x] = \int ...
0
votes
0answers
26 views

Question regarding expectation of log in EM

I am reading about the EM algorithm and I have 2 questions to ask. 1) The 1st line states that $ \sum p(x,z|\theta)] = $ the expectation of $ p(x|z)$. Is it because $E[f(x)] = xf(x) ?$ And because $...
1
vote
0answers
16 views

How to make sense of the EM algorithm expressed in terms of Kullback-Leibler divergence?

In the textbook by Wasserman, All of Statistics, the Algorithm is expressed as: 1) Pick a starting value $\theta^0$. 2) (E-Step). Calculuate: $$ J(\theta|\theta^j) = E_{\theta^j} \left(log \dfrac{f(...
2
votes
0answers
16 views

EM algorithm increase after E step?

It might be a silly question, but here it goes. The short version of my question is whether the marginal likelihood calculated after every E steps should be increasing or not. More details: Using ...
0
votes
0answers
33 views

understanding E step of EM for GMM

I'm reading this chapter about EM (9.3.1) of the book "Pattern Recognition and Machine Learning". I understand the basic EM algorithm for GMM, but I'm having some problems understanding the ...
0
votes
0answers
32 views

Expectation Maximization (EM) for 3-dimensional parameter $(\alpha,\mu_2,{\sigma}^2)$

Let $x_i$ where $i=1,...,100$ are iid observations from a mix of two normal distributions with means $\mu_1=0$ and $\mu_2$ and the same variance ${\sigma}^2$. If $\alpha$ is the proportion of the ...
0
votes
1answer
63 views

Compute smoothed probabilities for EM algorithm [closed]

In order to compute the expected value of log-likelihood in EM algorithm, we use 3 different probabilities Forecast (predictive) probabilities Inference probabilities Smoothed probabilities ...
1
vote
1answer
53 views

ZIP Fit Indices Calculated from an EM Algorithm

I am working through @ben-bolker's owls example available here:https://groups.nceas.ucsb.edu/non-linear-modeling/projects/owls/WRITEUP/owls.pdf In particular, I am making use of the R ...
0
votes
0answers
18 views

Is incremental EM a special case of SAGE?

I have come across "A View of the EM Algorithm that Justifies Incremental, Sparse, and Other Variants" (Neal & Hinton, 1998). I was wondering if incremental update in EM as described here can be ...
0
votes
0answers
11 views

How to ensure sample independence across iterations when using the EM algorithm?

I have a problem which uses the EM algorithm. For my purpose, I would like to run it such that the samples used in the $t$ iteration are independent from those used in the $t+1$ iteration. One way to ...
1
vote
0answers
52 views

observed data log likelihood and complete data log likelihood estimation

I have the following model. $p(z_{n}) = Categorical(\pi)$ $p(\pi) = Dir(\alpha)$ $p(x_{n}| z_{n}=k,\mu) = \prod_{d=1}^{D} {Bernouli(\mu_{kd})} $ $p(\mu_{kd}) = uniform(0,1)$ How can I find ...
0
votes
0answers
32 views

Fleshing out specific equation in “Introducing Monte Carlo Methods with R”

Page 157 of "Introducing Monte Carlo Methods with R" reads: A specific solution (Geyer and Thompson, 1992) consists in using only the conditional distribution $k(\pmb{ z }|\pmb{x},\hat{\theta}_{(...
0
votes
1answer
95 views

Log likelihood in EM Algorithm

I try understand the log likelihood in weka. I read about that is a probabilistic metric, but i cant understand, if is better when have low value or high value? How i can get the likelihood value, ...
1
vote
0answers
34 views

Maximum likelihood of coin toss of different type?

I was self-studying EM (Expectation Maximization) algorithm, where I came across this example given by the paper. In this paper, there are two types of coins A, B with unknown parameters $θ_A$ and $...
1
vote
1answer
62 views

Derivative in EM Algorithm?

The pic is from Andrew Ng's Machine Learning Class. It's about derivation in EM algorithm. I am not sure how to transfer the second line to the third line. Anyone has some good idea?
0
votes
0answers
34 views

Derivation of expectation maximization algorithm

I am reading this (http://ai.stanford.edu/~chuongdo/papers/em_tutorial.pdf) tutorial on expectation maximization algorithm, I am not able to understand second step. I have already read this (http://...
1
vote
0answers
89 views

How can I create a topic model with a mixture of multinomials and EM?

I'm trying to create a topic model with a mixture of multinomials and the EM algorithm. I do not want to use a package. For reference, I'm implementing this in Python with numpy. Data Sets I have ...
7
votes
1answer
209 views

Help in Expectation Maximization from paper :how to include prior distribution?

The Question is basd on the paper titled : Image reconstruction in diffuse optical tomography using the coupled radiative transport–diffusion model Download link The Authors apply EM algorithm with $...
0
votes
0answers
25 views

Approximating the conditional expectation in simulations

I am simulating stock returns, which are governed by the following equations $r_t = \mu + \delta r_{t-1} + \varepsilon_t$ $\sigma^2_t = \omega + \alpha \varepsilon_{t-1}^2 + \beta \sigma^2_{t-1}$ $\...
1
vote
1answer
60 views

I want to show a local optimum in my paper, how do I generate the data for it?

I'm writing a paper where I am explaining the problems of local optimum in my clustering algorithm. While clustering, in my data I would at times get local optimums. But I've tried and I cannot ...
1
vote
1answer
45 views

Clarification on EM Algorithm

So the general set-up for the EM algorithm is the following recursion \begin{align} \theta^{(t+1)}&=\text{argmax}_\theta\sum_zp(z\;|\;x,\theta^{(t)})\log\frac{p(x,z\;|\;\theta)}{p(z\;|\;x,\theta^{(...
0
votes
1answer
45 views

EM algorithm with dependent observations

I am trying to implement an EM algorithm for dependent observations. Specifically, I am dealing with families where the hidden variables $Z$ of the children are dependent on the hidden variables of ...
2
votes
0answers
37 views

What is a better political voting method? [closed]

We are in a season where some major elections are happening (e.g. U.S. elections) and I find it interesting to address. Objective When we decide "better", we need to define an objective. To be clear,...
1
vote
1answer
82 views

K-means algorithm's EM “Maximization” step

I'm a software engineer and am trying to understand how Lloyd's K-Means algorithm fits into the general framework of the Expectation-Maximization (EM) algorithm. I previously read the question "...
0
votes
1answer
34 views

Multinomial logistic regression where two choices are pooled/censored in the data?

I am looking for a lead on estimating a specific type of multinomial choice model. Specifically, assume that I see $N$ people and those people have some vector of characteristic $X$. For ten periods,...
2
votes
2answers
287 views

Initializing EM algorithm [duplicate]

I am using EM algorithm to estimate measured data ($y$) as a sum of two weighted gaussian distributions: $$model = \sum \limits_{i=1}^{L=2} w_i \phi(\theta_i)$$ Where $\theta$ = ($\mu$, $\sigma$). ...
0
votes
1answer
188 views

Interpreting Cluster Analysis from SAS Enterprise Miner

I am currently doing a text mining project and I conducted a clustering analysis in SAS enterprise miner. I am using the following settings: Anyway, The results look like this, showing me ...
2
votes
3answers
123 views

Is it possible to see the slow decreasing in test negative-log-likelihood as overfitting?

We have developed a model for some real data and we use EM algorithm for optimization of the model (parameters). In first phase we generate synthetic data according to the model (with some known ...
0
votes
0answers
49 views

Does the EM algorithm converge exactly where a grid search on the marginalized likelihood converges?

I have successfully implemented a grid search algorithm to estimate two parameters of a likelihood. I computed the likelihood $l(X;\theta)$ of observed data $X$ by integrating out the discrete ...