Skip to main content

Questions tagged [expectation-maximization]

An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.

Filter by
Sorted by
Tagged with
2 votes
1 answer
49 views

In Expectation-Maximization, in the maximization step, do we maximize expectation of the log likelihood (wikipedia) or evidence lower bound (cs 229)?

From cs 229 page 6: Intuitively, the EM algorithm alternatively updates Q and θ by a) setting Q(z) = p(z|x; θ) following Equation (8) so that ELBO(x; Q, θ) = log p(x; θ) for x and the current θ, and ...
figs_and_nuts's user avatar
0 votes
0 answers
19 views

How to evaluate this conditional expectation for the E-step in expectation-maximisation?

I'm trying to devise an expectation-maximisation algorithm for a certain problem but I'm unable to derive the conditional expectation in the E-step. For the purpose of this question I'll simplify the ...
DangerousTim's user avatar
2 votes
2 answers
62 views

Why does Variational Inference work?

ELBO is a lower bound, and only matches the true likelihood when the q-distribution/encoder we choose equals to the true posterior distribution. Are there any guarantees that maximizing ELBO indeed ...
Daniel Mendoza's user avatar
0 votes
0 answers
16 views

For EM algorithm, why we assign an independent distribution $Q_i$ for each sample index

I'm learning the expectation maximization algorithm with Andrew's CS229 lecture notes https://pillowlab.princeton.edu/teaching/statneuro2020/notes/notes18_LatentVariableModels.pdf The derivation and ...
BlackSith's user avatar
0 votes
0 answers
26 views

Can complicated $\mathcal{Q} (\theta; \theta^{\text{old}})$ function be replaced by log-likelihood when implementing/coding EM algorithm?

I am working on a MLE problem where one of the parameters does not have a closed-form solution. I have a proposal for $\theta^{\text{new}}$, but reject it if it does not improve $\mathcal{Q} (\theta; \...
Stew P. Doe's user avatar
3 votes
2 answers
48 views

Does Factor Analysis completely mitigate the singular covariance matrix problem?

Background I have been trying to understand Stanford CS 229’s lecture about Factor Analysis and the accompanying lecture notes. The lecturer introduced Factor Analysis as a way to mitigate the ...
fumoboy007's user avatar
1 vote
2 answers
41 views

Tractability of the E Step of the Expectation Maximization Algorithm

To optimize a probabilistic model with latent variables, I was trying to convince myself that there are situations where computing the marginalization $p(x)$ is intractable and computing the $Q(\theta ...
Lei Mao's user avatar
  • 61
1 vote
0 answers
80 views

Datasets with multiple maximum likelihood estimators

There is a sizeable body of literature on the issue of multiple maximizers in maximum likelihood estimation, such as https://projecteuclid.org/journals/statistical-science/volume-15/issue-4/...
Tom Solberg's user avatar
0 votes
0 answers
31 views

Under What Conditions Does a Gaussian Mixture Model (GMM) Have Maximum Entropy?

Introduction I'm delving into Gaussian Mixture Models (GMMs) within unsupervised learning frameworks and am particularly interested in their statistical properties, with a focus on entropy. Entropy ...
Alireza Ghazavi's user avatar
0 votes
0 answers
17 views

Expected Variance of EM Estimator in GMM with Respect to Observations

Title: Variance of EM Estimator in GMM with Respect to Observations Body: I'm estimating a parameter S from observations X and <...
Alireza Ghazavi's user avatar
1 vote
0 answers
40 views

Understanding Variational inference and EM in relation to each other

I have read several answers like here but, somehow I still have a few doubts. I hope to present my understanding and ask a few questions to clear my doubts EM: A maximization maximization algorithm E-...
figs_and_nuts's user avatar
4 votes
3 answers
159 views

Justification of independence assumption for latent variables in Expectation Maximization algorithm

When deriving the ELBO/free energy in the EM algorithm, it is often done in a "general" case of observed and latent variables and then an assumption of independent (or iid) variables is ...
user246795's user avatar
0 votes
0 answers
24 views

Can I assume that this is a GMM?

I'm trying to find the MLE for the parameters of the following distribution: $$f(x) = a \ \mathcal{N}(\mu_a, 1) + \beta \ \mathcal{N}(\mu_\beta, 1)$$ Taking the log likelihood of this complicates ...
John Katsantas's user avatar
0 votes
0 answers
32 views

Definition of expectation with condition variables

I am having a hard time of digesting this, which is part of EM algorithm that I borrowed Equation 3.2.7 from https://www.informit.com/articles/article.aspx?p=363730&seqNum=2#:~:text=3.2%...
JasonH's user avatar
  • 9
3 votes
0 answers
20 views

Feature importance in expectation maximization

The context is using EM algorithm for a mixture model - more precisely Dirichlet Multinomial Mixture, as discussed in Dirichlet Multinomial Mixtures: Generative Models for Microbial Metagenomics. One ...
Roger V.'s user avatar
  • 4,439
0 votes
0 answers
12 views

How to exploit existing closed-form update in stochastic coordinate descent?

I want to minimize a loss function $L(\Theta \mid X)$ given data $X = \{x_1,\dots,x_D\}$ where $D$ is large. The loss can be decomposed as: $L(\Theta \mid X) = \sum_{d=1}^D l(\Theta \mid x_d)$ where ...
Kaiwen's user avatar
  • 81
0 votes
1 answer
92 views

Proving that K-means corresponds to an EM algorithm?

Just wanted to make sure that my proof is correct and that I am not missing anything in the process. Any thoughts? " To demonstrate mathematically that the K-means algorithm corresponds to an ...
Naomi Pomella's user avatar
0 votes
0 answers
28 views

Should EM algorithm's final imputed mean match the initial parameter?

I am running a manual EM (expectation-maximization) algorithm in r. My code is the following: ...
flâneur's user avatar
1 vote
0 answers
42 views

Manually program EM in r to updated multiple parameters and solve missing data [closed]

I am trying to use EM (Expectation-maximization) to fill in missing data in R, but am not sure how to model/code it for my specific case. I am generally trying to follow the example format used in ...
flâneur's user avatar
0 votes
0 answers
73 views

How is the unigram tokenization using EM algorithm?

I intuitively understand what is happening in the unigram tokenizer and I think I also understand the EM algorithm if I can figure out the formulation in which I understand it i.e. What is the latent ...
figs_and_nuts's user avatar
0 votes
0 answers
37 views

Derivation of EM algorithm for Gaussian mixture

I am going through Expectation Maximization (EM) algorithm derivation for Gaussian Mixture models. I understand it except for a small detail. So, the general idea of EM is to maximize the expectation ...
baronett's user avatar
0 votes
0 answers
27 views

M-step of the EM algorithm for finding the ML estimate of θ

Can someone explain to me the procedure of computing the M step of the EM algorithm for a distribution that belongs to a regular exponential family? If I had a set of steps to follow I think I would ...
compscinewb's user avatar
1 vote
1 answer
50 views

How does expectation maximization relate to weighted least squares?

For the past few days I have been trying to implement the EM-algorithm in order to segment stores into k-clusters. What I already did was derivation of the complete-log-likelihood and also performed ...
Tim's user avatar
  • 273
0 votes
0 answers
27 views

Model fitting with Chinese Restaurant Process

I am trying cluster a trajectory, consisting of (state, action) sequences, by assigning them to the most likely model that generated them using Chinese Restaurant Process. Basically my goal is to ...
jumov's user avatar
  • 51
0 votes
1 answer
50 views

EM algorithm for mixture with latent regression?

I have in the past implemented the EM algorithm for certain cases of mixture distributions. However, I'm attempting to implement it now for a given problem that's exposing my lack of understanding of ...
statsplease's user avatar
  • 2,842
0 votes
0 answers
21 views

Calculate variance from the fixed parameters of a joint model for longitudinal data with informative times

I am trying to get the EM algorithm for a joint model for longitudinal data with informative time measurements. In the simpler case, I am considering the following model $$Y_{ij}=X_{ij}^T\gamma+U_i+Z_{...
adrimsvieira's user avatar
11 votes
2 answers
1k views

Why go through the trouble of expectation maximization and not use gradient descent?

In expectation maximization first a lower bound of the likelihood is found and then a 2 step iterative algorithm kicks in where first we try to find the weights (the probability that a data point ...
figs_and_nuts's user avatar
0 votes
0 answers
33 views

Why is evidence lower bound concave in the parameters?

The expectation maximization algorithm maximizes the evidence lower bound given by the following equation: $$ELBO = \Sigma_zQ(z)log\frac{p(x,z;\theta)}{Q(z)}$$ Now, the mechanism of the EM algorithm ...
figs_and_nuts's user avatar
4 votes
1 answer
665 views

Difference between KDE, MLE and EM for density estimation

I'm reviewing kernel density estimation (KDE), maximum likelihood estimation (MLE) and expectation maximization (EM) algorithm for density estimation and struggling to differentiate what each ...
Amith Adiraju's user avatar
2 votes
2 answers
223 views

MAP estimation for a Gaussian mixture using EM. Concerns with the covariance update formula

I am implementing the EM algorithm for a Gaussian mixture model with prior; that is, I am using the EM algorithm to find the MAP estimate, rather than the ML estimate. As briefly discussed in section ...
ummg's user avatar
  • 145
0 votes
0 answers
95 views

Expectation Maximization on Multivariate Gaussian Mixture Model for clustering

I have a dataset with 1000 observations and two features that define those N=1000 data points. Hence it is 1000*2 input matrix. I need to cluster them into k clusters. I am not understanding the E-M ...
Oindri's user avatar
  • 1
0 votes
0 answers
49 views

Q function in Baum-Welch algorithm

Assume there's a hidden Markov Model $\lambda=(\pi, \mathbf{A}, \mathbf{B})$, where $\pi$ is an initial distribution, $\mathbf{A}$ is a transition matrix and $\mathbf{B}$ is an emission matrix. Also, ...
thesecond's user avatar
  • 390
2 votes
1 answer
109 views

How can one estimate the parameters of unobserved variables?

Consider three discrete variables $X, Y , Z$ where $$X \sim B(p_x)$$ $$P(Y = 1 |X = 1) = p_y,\ P(Y = 0 |X = 1) = 1 - p_y,\ P(Y = 1 |X = 0) = 0$$ I think I can say that $$Y \sim B(p_xp_y)$$ And $$S = ...
Knz's user avatar
  • 23
3 votes
1 answer
50 views

Determine the parameters of a particle filter that best fit observations

I am wondering is there any established framework to optimize the parameter $\lambda$ of a particle filter such that $p(O|\lambda)$ is maximized, where $O$ is the observation sequence. For HMM and ...
Geng Wang's user avatar
1 vote
1 answer
87 views

The Tsallis entropy of generalized Gaussian distribution

I would like to discuss the computation of the Tsallis entropy for the generalized Gaussian distribution. From the paper in the link https://www.sciencedirect.com/science/article/pii/S0167947322000822....
M.cadirci's user avatar
0 votes
0 answers
91 views

EM algorithm get new parameters by optimizing the Q function (lower bound of likelihood function) or optimizing the likelihood function

We know that in the EM (Expectation-Maximization) algorithm, the E-step determines the $Q$ function by calculating expectations, which is a lower bound of the likelihood function. In the M-step, by ...
yi ren's user avatar
  • 1
0 votes
0 answers
118 views

How to compute the covariance matrix for a mixture model estimated by the EM algorithm

I am trying to compute the observed Fisher information matrix for a mixture model estimated by the EM algorithm. My original thought is to simply compute the second derivative of a mixture density. ...
Lydia2kkx's user avatar
0 votes
0 answers
19 views

number of parameters in Dirichlet Mixture Model clustering (non-bayesian)

I made a function that implements the clustering algorithm in the research article "Clustering compositional data using Dirichlet mixture model" (2022). I am now trying to figure out which ...
Immanuel Kunt's user avatar
1 vote
0 answers
31 views

Which Fisher information to use to obtain Cramer-Rao bound in expectation-maximization?

I have a rather limited understanding of statistical estimation theory so I apologize if my question is strange or trivial. Say I have an expectation-maximization-based algorithm for determining the ...
DangerousTim's user avatar
0 votes
0 answers
63 views

EM algorithm "maximization step" for product of two independent Bernoulli variables

We are given the following model, where the event $C=1$ is defined using two other Bernoulli variables $E$ and $R$: $$P(C=1|q,d,k)=P(E=1|k)\cdot P(R=1|q,d)$$ Which we denote as: $$\theta_k=P(E=1|k), \...
Noral Gata's user avatar
0 votes
1 answer
112 views

Fitting a Gaussian Mixture Model with known share of noise/outliers

A Gaussian Mixture model is fitted by the Expectation-Maximization algorithm. This fairly simple iterative algorithm consists of two steps and the initialization. Initialization (for ...
Klops's user avatar
  • 156
1 vote
1 answer
301 views

Why complete data log likelihood in M-step of EM algorithm

In Bishop's Pattern Recognition and Machine Learning book(page440), it talks about the M-step in EM algorithm of Gaussian Mixture Model. I am confused about the likelihood function of M-step. By ...
user3153824's user avatar
0 votes
2 answers
109 views

How to understand the binary latent variable z in GMM model?

GMM(Gaussian Mixture Model) itself is a mixture of Gaussian with each having the proportion of $\pi_k$, $$\sum_{k=1}^{K}\pi_k=1$$this is easy to understand. But when introducing the latent, I don't ...
user3153824's user avatar
0 votes
0 answers
23 views

Understanding Monte Carlo EM in a mixed effects model context

I am currently reading Mixed Effects Models for Complex Data by Lang Wu. On page 136, the author mentions the Monte Carlo EM algorithm as a way of treating missing covariate in the mixed effect model. ...
Faithhhhhh's user avatar
0 votes
1 answer
233 views

Why is the E-step in the EM algorithm called this way?

Yes, this has been asked before here, but for different reasons. In the E-Step nothing is calculated, we simply define the function, yet once it is defined it is defined once and for all. We could ...
timtam's user avatar
  • 203
1 vote
0 answers
111 views

Is the EM algorithm guaranteed to converge if the log likelihood is concave

As the EM algorithm is guaranteed to increase the log likelihood at each iteration. If the log likelihood is concave is it guaranteed to converge to the maximum of the likelihood, that is will we get ...
Dylan Dijk's user avatar
1 vote
1 answer
73 views

How should I analyze longitudinal data with missing observations for individuals? Expectation Maximization or panel data analysis or regression? How?

I possess a dataset containing 1,000 chickens. For each chicken, we have egg count data for various sizes (11 to 22mm) and the overall number of chicks that hatched from these eggs for each chiken. ...
insan's user avatar
  • 145
2 votes
0 answers
19 views

MNAR Imputation On Time Series Data in Ongoing Study

I am working on an ongoing study that tracks subjects through time. Each subject is enrolled for months, where upon exiting I calculate their length-of-stay. Since the study is ongoing, I have both ...
MVH's user avatar
  • 21
2 votes
1 answer
73 views

MLE for a mixture of betas without using EM algorithm

Suppose I have a mixture of two Beta densities say $f_1 = \text{Beta}(1,1)$ and $f_2= \text{Beta}(1,\beta)$ where $\beta$ is unknown. The sample $X_1,....,X_n$ is observed based on latent Bernoulli ...
Maths Freak's user avatar
0 votes
0 answers
33 views

Can we get a Global Maxima by using EM algorithm? [duplicate]

Actually, I read many articles and blogs that EM converges to local maxima can someone help me that it converges to local maxima always? Can we get a global maximum?
RAJ SHIVAM's user avatar

1
2 3 4 5
13