# Questions tagged [expectation-maximization]

An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.

607 questions
Filter by
Sorted by
Tagged with
49 views

### In Expectation-Maximization, in the maximization step, do we maximize expectation of the log likelihood (wikipedia) or evidence lower bound (cs 229)?

From cs 229 page 6: Intuitively, the EM algorithm alternatively updates Q and θ by a) setting Q(z) = p(z|x; θ) following Equation (8) so that ELBO(x; Q, θ) = log p(x; θ) for x and the current θ, and ...
• 2,603
19 views

### How to evaluate this conditional expectation for the E-step in expectation-maximisation?

I'm trying to devise an expectation-maximisation algorithm for a certain problem but I'm unable to derive the conditional expectation in the E-step. For the purpose of this question I'll simplify the ...
62 views

### Why does Variational Inference work?

ELBO is a lower bound, and only matches the true likelihood when the q-distribution/encoder we choose equals to the true posterior distribution. Are there any guarantees that maximizing ELBO indeed ...
16 views

### For EM algorithm, why we assign an independent distribution $Q_i$ for each sample index

I'm learning the expectation maximization algorithm with Andrew's CS229 lecture notes https://pillowlab.princeton.edu/teaching/statneuro2020/notes/notes18_LatentVariableModels.pdf The derivation and ...
26 views

• 61
1 vote
80 views

### Datasets with multiple maximum likelihood estimators

There is a sizeable body of literature on the issue of multiple maximizers in maximum likelihood estimation, such as https://projecteuclid.org/journals/statistical-science/volume-15/issue-4/...
• 139
31 views

### Under What Conditions Does a Gaussian Mixture Model (GMM) Have Maximum Entropy?

Introduction I'm delving into Gaussian Mixture Models (GMMs) within unsupervised learning frameworks and am particularly interested in their statistical properties, with a focus on entropy. Entropy ...
17 views

### Expected Variance of EM Estimator in GMM with Respect to Observations

Title: Variance of EM Estimator in GMM with Respect to Observations Body: I'm estimating a parameter S from observations X and <...
1 vote
40 views

### Understanding Variational inference and EM in relation to each other

I have read several answers like here but, somehow I still have a few doubts. I hope to present my understanding and ask a few questions to clear my doubts EM: A maximization maximization algorithm E-...
• 2,603
159 views

### Justification of independence assumption for latent variables in Expectation Maximization algorithm

When deriving the ELBO/free energy in the EM algorithm, it is often done in a "general" case of observed and latent variables and then an assumption of independent (or iid) variables is ...
• 101
24 views

### Can I assume that this is a GMM?

I'm trying to find the MLE for the parameters of the following distribution: $$f(x) = a \ \mathcal{N}(\mu_a, 1) + \beta \ \mathcal{N}(\mu_\beta, 1)$$ Taking the log likelihood of this complicates ...
32 views

### Definition of expectation with condition variables

I am having a hard time of digesting this, which is part of EM algorithm that I borrowed Equation 3.2.7 from https://www.informit.com/articles/article.aspx?p=363730&seqNum=2#:~:text=3.2%...
20 views

### Feature importance in expectation maximization

The context is using EM algorithm for a mixture model - more precisely Dirichlet Multinomial Mixture, as discussed in Dirichlet Multinomial Mixtures: Generative Models for Microbial Metagenomics. One ...
• 4,439
12 views

### How to exploit existing closed-form update in stochastic coordinate descent?

I want to minimize a loss function $L(\Theta \mid X)$ given data $X = \{x_1,\dots,x_D\}$ where $D$ is large. The loss can be decomposed as: $L(\Theta \mid X) = \sum_{d=1}^D l(\Theta \mid x_d)$ where ...
• 81
92 views

### Proving that K-means corresponds to an EM algorithm?

Just wanted to make sure that my proof is correct and that I am not missing anything in the process. Any thoughts? " To demonstrate mathematically that the K-means algorithm corresponds to an ...
28 views

### Should EM algorithm's final imputed mean match the initial parameter?

I am running a manual EM (expectation-maximization) algorithm in r. My code is the following: ...
• 65
1 vote
42 views

### Manually program EM in r to updated multiple parameters and solve missing data [closed]

I am trying to use EM (Expectation-maximization) to fill in missing data in R, but am not sure how to model/code it for my specific case. I am generally trying to follow the example format used in ...
• 65
73 views

### How is the unigram tokenization using EM algorithm?

I intuitively understand what is happening in the unigram tokenizer and I think I also understand the EM algorithm if I can figure out the formulation in which I understand it i.e. What is the latent ...
• 2,603
37 views

### Derivation of EM algorithm for Gaussian mixture

I am going through Expectation Maximization (EM) algorithm derivation for Gaussian Mixture models. I understand it except for a small detail. So, the general idea of EM is to maximize the expectation ...
• 11
27 views

### M-step of the EM algorithm for finding the ML estimate of θ

Can someone explain to me the procedure of computing the M step of the EM algorithm for a distribution that belongs to a regular exponential family? If I had a set of steps to follow I think I would ...
1 vote
50 views

### How does expectation maximization relate to weighted least squares?

For the past few days I have been trying to implement the EM-algorithm in order to segment stores into k-clusters. What I already did was derivation of the complete-log-likelihood and also performed ...
• 273
27 views

### Model fitting with Chinese Restaurant Process

I am trying cluster a trajectory, consisting of (state, action) sequences, by assigning them to the most likely model that generated them using Chinese Restaurant Process. Basically my goal is to ...
• 51
50 views

### EM algorithm for mixture with latent regression?

I have in the past implemented the EM algorithm for certain cases of mixture distributions. However, I'm attempting to implement it now for a given problem that's exposing my lack of understanding of ...
• 2,842
21 views

• 23
50 views

### Determine the parameters of a particle filter that best fit observations

I am wondering is there any established framework to optimize the parameter $\lambda$ of a particle filter such that $p(O|\lambda)$ is maximized, where $O$ is the observation sequence. For HMM and ...
1 vote
87 views

### The Tsallis entropy of generalized Gaussian distribution

I would like to discuss the computation of the Tsallis entropy for the generalized Gaussian distribution. From the paper in the link https://www.sciencedirect.com/science/article/pii/S0167947322000822....
91 views

### EM algorithm get new parameters by optimizing the Q function (lower bound of likelihood function) or optimizing the likelihood function

We know that in the EM (Expectation-Maximization) algorithm, the E-step determines the $Q$ function by calculating expectations, which is a lower bound of the likelihood function. In the M-step, by ...
118 views

### How to compute the covariance matrix for a mixture model estimated by the EM algorithm

I am trying to compute the observed Fisher information matrix for a mixture model estimated by the EM algorithm. My original thought is to simply compute the second derivative of a mixture density. ...
19 views

### number of parameters in Dirichlet Mixture Model clustering (non-bayesian)

I made a function that implements the clustering algorithm in the research article "Clustering compositional data using Dirichlet mixture model" (2022). I am now trying to figure out which ...
1 vote
31 views

### Which Fisher information to use to obtain Cramer-Rao bound in expectation-maximization?

I have a rather limited understanding of statistical estimation theory so I apologize if my question is strange or trivial. Say I have an expectation-maximization-based algorithm for determining the ...
63 views

23 views

### Understanding Monte Carlo EM in a mixed effects model context

I am currently reading Mixed Effects Models for Complex Data by Lang Wu. On page 136, the author mentions the Monte Carlo EM algorithm as a way of treating missing covariate in the mixed effect model. ...
• 323
233 views

### Why is the E-step in the EM algorithm called this way?

Yes, this has been asked before here, but for different reasons. In the E-Step nothing is calculated, we simply define the function, yet once it is defined it is defined once and for all. We could ...
• 203
1 vote
111 views

### Is the EM algorithm guaranteed to converge if the log likelihood is concave

As the EM algorithm is guaranteed to increase the log likelihood at each iteration. If the log likelihood is concave is it guaranteed to converge to the maximum of the likelihood, that is will we get ...
• 359
1 vote
73 views

### How should I analyze longitudinal data with missing observations for individuals? Expectation Maximization or panel data analysis or regression? How?

I possess a dataset containing 1,000 chickens. For each chicken, we have egg count data for various sizes (11 to 22mm) and the overall number of chicks that hatched from these eggs for each chiken. ...
• 145
19 views

### MNAR Imputation On Time Series Data in Ongoing Study

I am working on an ongoing study that tracks subjects through time. Each subject is enrolled for months, where upon exiting I calculate their length-of-stay. Since the study is ongoing, I have both ...
• 21
Suppose I have a mixture of two Beta densities say $f_1 = \text{Beta}(1,1)$ and $f_2= \text{Beta}(1,\beta)$ where $\beta$ is unknown. The sample $X_1,....,X_n$ is observed based on latent Bernoulli ...