Questions tagged [expectation-maximization]

An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.

170 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6
votes
1answer
148 views

Does marginalization of some of the latent variables improve convergence in EM?

Given a likelihood to maximize $$ \log p(x | \theta) $$ Imagine that, in order to apply EM, we can augment the model with one or two latent variables. In that case, we can derive two lower bounds: $...
6
votes
0answers
648 views

Likelihood maximization: MCEM algorithm versus MCMC algorithm

Hello Everyone this is my first question. I am a particle physicist and I am doing some empirical studiues on parameters estimation using different methods (this might give me some handle to study on ...
6
votes
0answers
146 views

Is this problem Bayesian? And can I use variational approximation?

Suppose there are $N$ samples of observations $\mathbf X(n)$ ($n=1,\cdots,N$), which are given by probability distribution $p(\mathbf X(n)|\mathbf Z(n))$ with their conditions are given by hidden ...
6
votes
0answers
610 views

Computing Standard Errors in EM algorithm

I'm applying the EM to a hidden markov chain (the $\mathbf{Z}=\{Z_1,...,Z_n\}$ variable), with observations(the $\mathbf{Y}=\{Y_0,...,Y_n\}$ variable) dependent not only on the hidden markov chain, ...
5
votes
0answers
63 views

How to choose $k$ of $n$ urns to maximize the number of colors of balls?

Say we have $N$ urns, each containing many colored balls, but we only know the probability $p_{ci}$ that urn $i$ contains at least one ball of color $c$ (expressible as a $c$-vector ${\bf p}_i$). ...
4
votes
0answers
39 views

Expectation Maximisation (EM) Algorithm

Some of my parameters do not have a closed form solution. Thus, for these parameters the M-step is implemented via a one-step Newton-Raphson update, i.e., \begin{equation} \theta^{t+1} = \theta^t - \...
4
votes
0answers
64 views

Implementation of EM algorithm confusion

Here EM algorithm manually implemented, there's a question of the implementation in R of the EM algorithm for 2 mixed gaussians. The answer has a supposedly correct implementation. However, don't the ...
4
votes
0answers
672 views

Fisher information matrix in logistic regression

I am self-studying the basics of logistic regression. I came across this sentence: In logistic regression expected and observed information matrixes are equal I am aware that the information ...
4
votes
1answer
429 views

Expectation Maximization intuitive explanation

Given a set of events {A, B, C, D, E} that occur once each month for n years: ...
4
votes
0answers
86 views

Interesting application of E-M algorithm

Suppose the following dataset: [3 4 3 4 6 12 12 7 8 9] [2 5 3 4 12 2 2 10 7 6] [3 4 3 4 5 11 10 7 8 9] These numbers are totally random. So this dataset, depicts ...
4
votes
0answers
1k views

Expectation-Maximization with a MLE package in R

As a follow up to one answer of the topic Expectation-Maximization with a coin toss: One of the user posted an R-code with MLE example almost a year ago (and his last online time here was 3 months ago,...
4
votes
1answer
64 views

EM algorithm increase after E step?

It might be a silly question, but here it goes. The short version of my question is whether the marginal likelihood calculated after every E steps should be increasing or not. More details: Using ...
4
votes
0answers
100 views

Averaged estimators in stochastic versions of EM

Recently I've been working EM algorithms for MAP estimation in a problem where the expectation is intractable, but the maximization is easy. Further, draws from the distribution in the E-step are ...
3
votes
0answers
148 views

Termination Condition(s) for Expectation Maximization

What are good criteria for deciding when to terminate the expectation-maximization algorithm. I know that the idea is that you should terminate when the change in the data log likelihood is "small" ...
3
votes
2answers
4k views

Simple Explanation of Baum Welch/Viterbi

I'm looking for a very simple explanation as possible for Baum Welch and Viterbi for HMMs with a straightforward well annotated example. Almost all of the explanations I find on the net invariably ...
3
votes
0answers
264 views

Question about E step in Em algorithm, a challenge part, any help please?

I am new to EM algorithm and copula. I was reading a paper in mixture pair-copula. The authors use $u=(u_r, u_s) = (u_r^t,u_s^t), (t= 1,...,T)$ to indictae to the vector of copula data. Then, they ...
3
votes
0answers
211 views

Convergence of k-means or EM on Mixture of Gaussians

There are many algorithms for learning mixture of Gaussians but typically k-means/EM is used in practice. My question is related to the performance of k-means/EM for MoG. Recently, I came across this ...
3
votes
0answers
73 views

Indirect solution for maximum entropy through sampling?

Is there a way to sample from a finite set $\{A,B,C,D\}$ such that the limiting empirical proportions converges to the maximum entropy solution of their probabilities consistent with known constraints?...
3
votes
0answers
89 views

Cross validation or EM for selecting strength of the prior?

Often when I'm looking at bayesian analyses, the influence of the prior is chosen via cross validation. For example, suppose $X$ and $Y$ represent some real valued data that I want perform a bayesian ...
3
votes
0answers
239 views

Why isn't a gaussian mixture prone to overfitting?

Consider a Gaussian mixture of 2 components and a dataset of size $N$. The EM algorithm use the data to estimate: the model parameters: the means $\mu_1, \mu_2$ (say the covariances matrices are ...
3
votes
0answers
816 views

EM algorithm: With prior vs. not prior

I have a working EM algorithm without prior. I am asking for some advice on how to add prior on latent variables. Define: $t_i \in \{ +1, -1 \} $: variables of interest to be predicted $p_j \in [0,...
3
votes
0answers
615 views

why is the incomplete log-likelihood difficult to optimize

I am trying to teach myself the expectation-maximization algorithm and the texts say the EM is particularly useful when the incomplete log-likelihood i.e. $P(X|\theta)$ where $\theta$ are the ...
3
votes
0answers
233 views

Expectation-Maximization with dependent latent variables

Deriving the equations for a Expectation Maximization over my model, I end up with a posterior for the latent variables (E-step) that prevents me from going on. Generative model I assume my data is ...
3
votes
0answers
541 views

Comparison of Variational Bayes and Expectation Maximization algorithms

I need to learn both the VB and EM methods for Bayesian Networks. Before going into detail of both algorithms, which I am a bit aware of, I need to EXACTLY understand the basic motivations behind them....
3
votes
0answers
745 views

EM algorithm for mixture of Negative Binomial distributions

I am trying to derive the EM-algorithm of mixtures of negative binomial distribution $Neg\;Bin(r,p)$. I have the updating equations for updating the E-step as well as $p$ and the mixing coefficients $\...
3
votes
0answers
131 views

Can we reconstruct the hidden (latent) variables after executing EM?

The question is in the title. I know that EM algorithm could do maximum likelihood estimation for models that have latent variables. I would like to know can we get the (estimated) value of these ...
2
votes
0answers
71 views
+250

Does EM algorithm require us to know the joint (predictive) distribution of the latent variables $Z$ when $Z$ is two-dimensional?

In its general form the E-step of the EM algorithm finds the expectation $$ Q(\theta|\theta') =\int \log[ p(Y,Z | \theta)] p(Z|Y,\theta') d Z$$ where $Y$ the data, $Z$ the latent variables, $\theta'$...
2
votes
0answers
18 views

Can someone verify if the following Bayesian Information Criterion (BIC) model selection algorithm is correct for Gaussian mixture models?

I am trying to find an automated way of picking the number of clusters $K \in \mathbb{N}$ for unsupervised learning scenarios, specifically for GMM. I was suggested to use something called the "...
2
votes
1answer
111 views

MCAR, MAR and EM

I have a binary(1/0) classification task. I am trying to find $p(y = 1 | X)$ where $X$ is the vector of input variables and $y$ is the binary output label. Suppose that for some records the output ...
2
votes
0answers
37 views

EM algorithm for mixture models, what happens when you don't know how many mixtures you have?

i am trying to learn something here so i tried to estimate the means of two normal distributions that I created. Here is the code: ...
2
votes
0answers
47 views

Is Expectation-Maximization the right type of analysis (& if so, how)?

I have 118 microbes that I have tested at various concentrations of a drug that is supposed to kill them (0.1875 - 15). However, these microbes have mutations (A-H) which confer resistance to the ...
2
votes
0answers
33 views

Multivariate mixture models

I am new to mixture modeling and have successfully used bernoulli mixture models to cluster datasets of binary data. My real purpose, though is to cluster datasets with mixed data types: normal, ...
2
votes
0answers
390 views

Using Naive Bayes classifier for unsupervised learning

I was going through this article to learn about how the EM algorithm can be used to use the Naive Bayes algorithm for unsupervised learning. Suppose we have the following data without labels: 1 0 1 1 ...
2
votes
3answers
533 views

GMM EM algorithm complexity per iteration

I was fitting GMM clusters with diagonal covariance on my data using EM with $n$ (=5e6) points, each having $m$ (=160) ...
2
votes
0answers
409 views

loglikelihood decrease very slightly in EM algorithm

I am working with a very large and complicated function. I am using EM algorithm to estimate the model parameters. The EM works very well. However, after 27 iteration I see that the values of the ...
2
votes
0answers
114 views

What is the result for EM algorithm in smoothed LDA model?

In the original LDA paper (Blei2003), EM algorithm estimates $\alpha$ and $\beta$ in Fig.5. So, what is the result for Fig.7? Will it give estimation of $\alpha, \beta$ or $\alpha, \eta$? And, if I ...
2
votes
0answers
52 views

Implementing a robit model

I am trying to implement a robit model. The answer here has a description. I currently have functioning code (written in R) that follows the procedure laid out in this paper. It looks like this: <...
2
votes
1answer
233 views

Another EM-algorithm problem

I have the following problem: I have a random vector $y$ which has length $l$. The first $z$ bits come from a Bernoulli random variable with parameter $\theta_1$ and the next $l-z$ come from a ...
2
votes
0answers
658 views

HMM (Baum-Welch) - convergence rate differences between the transition and output matrices

I am trying to learn more about the convergence properties of the Baum-Welch algorithm for estimating the HMM parameters. I ran a test comparing the convergence of both the transition and output ...
2
votes
0answers
361 views

How to ensure covariance matrix is positive semi definite in linear dynamical model learning?

I am trying to learn a linear dynamical model for a data using expectation-maximization algorithm. The model is defined as follows: $$x_0 \sim \mathcal{N}(\mu_0 ,\Sigma_0)$$ $$ x_{t+1} = Fx_t + w_t, \...
2
votes
0answers
705 views

Using HMM or depmixS4 package to find log-likelihood values

I am trying to implement a Hidden Markov Model. In my studies we used the package HMM as well as wrote our own functions. Here is a slight modification of the example from the HMM package. ...
2
votes
0answers
1k views

Entropy of a set of categorical variables

In the context of Expectation-Maximization, I would like to compute te entropy factor in order to get the value of the lower bound when the algorithm converged. This lower bound can be expressed as: ...
2
votes
0answers
46 views

mixed noise and gaussian

I have a large number of data sets. Each data set has something 200K data points lying in a square times a circle. The square is solid $I\times I$. The circle $S^1$ is hollow (dim 1). By reasoning ...
2
votes
0answers
352 views

Baum-Welch algorithm variation for Hidden Markov model with reward

Following my previous question on the subject I would like to get your feedback on the following alternative solution. (The original solution to this question is the usage of the POMDP model proposed ...
2
votes
0answers
120 views

Expectation Maximization Gaussian Mixture Example

I am a biologist trying to understand expectation maximization for a mixture of two Gaussian distributions. I think I understand how to deal with the means of the two distributions, but I don't know ...
2
votes
0answers
227 views

Confusion with EM Algorithm for Gaussian Mixture?

I am trying to learn EM Algorithm for Gaussian Mixture. But not able to understand few stuffs. This is what I have understood. Consider GMM with k components. $$ p( \mathbf{x}| \mathbf{\alpha_{k}},\...
2
votes
0answers
559 views

Numerical Approxmation of standard errors for parameter estimation in the EM algorithm

Generally, when you want to compute standard errors for estimated parameters within the ML framework, one uses the diagonal elements of the observed information matrix. In for instance ...
2
votes
0answers
432 views

How do you do EM algorithm for a factored model for a recommender system?

Let $X$ be a $n \times d$ matrix with users as rows and movies as columns. Each user is a single row $x^{(u)} \in \mathbb{R}^d$ (i.e. for user u there are at most d ratings for the d movies). Also ...
2
votes
0answers
79 views

Baum Welch and a 1 state Markov model?

I'm using the Baum-Welch algorithm to determine the parameters of a 2 state Hidden Markov Model. It determines fairly well. When I increase the sample size, the estimations get more concentrated, and ...
2
votes
1answer
437 views

Gaussian clusters and original distributions

In Gaussian clustering (i.e. General Mixture Models) we model the data with some clusters. For example, in the below figure, we have two clusters $C_1, C_2$, each of which are modeled with a Gaussian (...