Questions tagged [expectation-maximization]
An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.
568
questions
0
votes
0
answers
9
views
Hierarchical Expectation Maximization
Let us assume we have a generative model $y = g(\theta,x)$, where $\theta$ is a set of parameters, and $x$ is a set of latent variables. Given a datapoint $y$, I need to update the parameters $\theta$ ...
- 315
1
vote
0
answers
12
views
Kinds of Expectation-Maximization [closed]
I'm starting to dig into the variational inference literature, as I need to solve a learning problem where the latent variables form an hierarchical structure (an hierarchical Gaussian generative ...
- 315
2
votes
1
answer
38
views
Maximum likelihood for mixture of Bernoullis with known mixture proportions
Given the hierarchical model
$$
\begin{align}
k & \sim \text{Categorical}(\pi_1, \dots, \pi_K) \\
X \mid k & \sim \text{Bernoulli}(\theta_{k})
\end{align}
$$
and an i.i.d. sample $X_1, \dots, ...
- 3,257
1
vote
0
answers
14
views
How to get (Gaussian) mean and covariance of missing variables in the E-Step of the EM Algorithm for Missing Data
The M step for missing data imputation using MVN updates
$$\mu^t=\frac{1}{N_D}\sum_n E[y_n]\\
\Sigma^t=\frac{1}{N_d}\sum_nE[y_n y_n^\top]-\mu^t(\mu^t)^\top
$$
where $y_n=(x_n,z_n )$.
However, the ...
- 497
1
vote
0
answers
15
views
EM equality constraint using Jensen's Inequality
I'm trying to derive the ELBO as per CS229 Page 144, Eq 11.8.
For E-step, the LHS and RHS must be equal:
$$
\begin{align}
f\left(\mathbb{E}_{z\sim Q}\left[ \frac{p(x, z; \theta)}{Q(z)}\right]\right) \...
- 111
0
votes
0
answers
26
views
In what situations can you use EM, but where you cannot compute the marginal likelihood?
Expectation Maximisation is used to find the parameters of a hierarchical model with some nuisance parameters, that need to be integrated out. The typical example is a Gaussian Mixture Model, where ...
- 101
0
votes
0
answers
13
views
Observation from each cluster from EM algorithm manually implemented
I am interesting in implementing the EM algorithm manually. And I did it. However, I am still do not know how to get the observation from each cluster? For example, after run the EM algorithm, the ...
- 507
3
votes
1
answer
73
views
Need help understanding why the original proof of the EM algorithm had a flaw in it
I understand that in the original paper on the EM algorithm by Dempster, the proof of convergence in Theorem 2 had a flaw, and that a full proof wasn't given until CF Jeff Wu published it years later.
...
- 2,315
0
votes
0
answers
24
views
Continuously running the M step while approximating the E step
Let us assume we need to maximize the likelihood $p(x|\theta)$ of an observed variable $x$ given a set of parameters $\theta$. However, this likelihood also depends on a set of hidden variables $z$, ...
- 315
1
vote
1
answer
33
views
details of stochastic Expectation Maximization (EM) algorithm
I went into a paper, Online EM Algorithm for Latent Data Models (Olivier Cappé & Eric Moulines, 2009). I got confused by the first equation the authors wrote, the Q function:
Here the authors ...
- 11
0
votes
0
answers
20
views
Can overlap impact the EM algorithm?
I am using EM algorithm for clustering. I see that if there is an overlap, the accuracy is reduced. However, it is improvesd if no overlap exists.
My question is, can overlap affect the performance of ...
- 507
5
votes
0
answers
65
views
How does the EM algorithm work in Bayesian regression? [closed]
I have problem distinguishing between the latent variables $z_i$ and the parameters $\theta_i$ in EM algorithm. Suppose we have the hierarchical priors
\begin{aligned}
\beta|\tau,\omega &\sim \...
- 189
2
votes
0
answers
45
views
E-Step of the EM algorithm on mixture
Im trying to implement the EM algorithm on mixture model: $pg(x) + (1-p)h(x)$ where the sample $\bar{x} = (x_1, \ldots, x_n)$, is independently generated from the mixture, and $g(x) = e^{-x}$ and $h(x)...
- 119
2
votes
1
answer
60
views
EM algorithm on discrete random variable
Im trying to find the iteration formula for the maximum likelihood estimator of the parameter $\theta$ of the following discrete random variable, through the EM algorithm:
$$P(Y = y ) = \sum_{i=0}^{\...
- 119
0
votes
0
answers
14
views
In stochastic maximum likelihood and contrastive divergence, why do we sample from model distribution for partition function?
I have been reading the "Deep Learning" book from Ian Goodfellow. In a topic on Restricted Boltzmann Machines and how to train them, techniques like Stochastic Maximum Likelihood and ...
- 121
0
votes
1
answer
37
views
What is 'factored models'?
What is 'factored models'?
I've seen the term 'factored models' in paper 'The Bayesian Structural EM Algorithm'.
But I see it for the first time and it is hard to understand...
I tried searching it ...
- 43
0
votes
0
answers
36
views
Solve the EM algorithm of the Gaussian mixture model
Considering a mixture of two normally distributed random variables $Y_1$ and $Y_2$ and considering the linear combination of the two:
$$
\begin{gathered}
Y_{1} \sim N\left(\mu_{1}, \sigma_{1}^{2}\...
- 609
0
votes
0
answers
16
views
Expectation Maximization: How to perform "E" step in coin flipping example? [duplicate]
I recently read this primer that does a great job of explaining the principles of the Expectation-Maximization (EM) "algorithm."
However, I'm confused about how they calculated the ...
- 211
0
votes
0
answers
75
views
Hidden Markov Model - Baum Welch algorithm initialization
Currently I'm working on a problem where I have a multidimensional, continuous sequence of observations $X$ that model my response variable $y$ with two states $0$ and $1$. I assume that this sequence ...
- 279
0
votes
0
answers
29
views
How to conduct EM algorithm when there are some outliers in GMM Models?
I'm just confused about the problem of adding an outlier component directly to the primary form of GMM models:
Suppose that the observed data contains several outliers. The mixture model could be:
$$
...
- 1
0
votes
1
answer
36
views
Conditional independence in EM algorithm
Let $X$, $\theta$ and $Z$ denote observed, parameter and latent nodes in a graphical model. The EM algorithm attempts to find a local maximum likelihood estimate $\theta^\ast$ for the likelihood of ...
- 211
3
votes
1
answer
137
views
Is the likelihood for Gaussian mixture models still multimodal when Y is partially observed?
In discussing Gaussian mixture models (GMMs), https://normaldeviate.wordpress.com/2012/08/04/mixture-models-the-twilight-zone-of-statistics/ highlights the issue of
Multimodality of the Likelihood. ...
- 3,850
1
vote
1
answer
42
views
Issue with Casella&Berger derivation of EM likelihood equality
In the explanation of the EM (Expectation maximization) algorithm p.328 in the book "Statistical inference" by G. Casella and R. Berger, 2nd edition, they present the following:
$\mathbf{Y} =...
- 367
0
votes
0
answers
17
views
Expectation-Maximization high missing rates and multiple variables
I know MICE can be used for imputation of multiple variables simultaneously. The expectation maximization approach (EM) can be used to impute missing data. Typically, one should only be using ...
- 83
0
votes
0
answers
42
views
Labeling unlabeled data (expectation maximization)
Say I have a database (Excel) consisting of 10k different dresses and accompanying attributes (column names) for each dress (sleeve length, color, pattern, ...). I would like to label each of these ...
0
votes
1
answer
151
views
How does expectation maximization compute missing data?
I have been searching for a simple example of how expectation-maximization (EM) computes missing data. All the examples I have found are based on multivariate normal models. I have seen that EM can be ...
- 13
1
vote
0
answers
89
views
Problems with convergence of the EM algorithm for a gaussian mixture regression
I have been implementing a EM-algorithm for a latent-class regression model, where every individual has a vector of observations.
Currenly, I have the problem that the model does not converge. The log ...
- 21
0
votes
0
answers
90
views
EM derivation for mixture of t distributions
(Notation is at the bottom.)
I'm currently self-studying the Expectation-Maximization algorithm and has learned to derive it for various mixture models including Gaussian mixture models, mixture of ...
- 124
0
votes
0
answers
10
views
Model choice with expectation maximization: which likelihood?
When deciding about the number of mixture components using Akaike or bayesian information criteria, should one use the full likelihood or the likelihood marginalized over the latent variables? Both ...
- 1,800
0
votes
0
answers
70
views
Benefits of Expectation Maximization for Mixture Models
What are the benefits of using expectation maximization for mixture models vs. direct maximization of the marginal likelihoods?
Analytic maximization step
In case of Gaussian mixtures the benefit is ...
- 1,800
0
votes
1
answer
94
views
Computing a prior from two components in Naive Bayes
Given a model parameter $\theta$ that is composed of two distributions in a Naive Bayes classifier, how is $P(\theta)$ typically computed in practice?
More specifically, from the article of Nigam et ...
- 230
0
votes
0
answers
84
views
Expectation maximization: does the likelihood always increase monotonically?
When working with (gaussian) mixture models, I always took it for a mathematical fact that the marginal likelihood increases with every iteration step. If it were not the case, it always meant an ...
- 1,800
0
votes
0
answers
82
views
EM algorithm for multinomial data
Good afternoon ,
Assume we have a dataset modeled by multinomial data of K cluster.
To apply EM algorithm parameters are :
$${\theta }^{\left(0\right)}=({p}_{k}^{\left(0\right)},k=1,\mathrm{\dots },K)$...
- 113
2
votes
0
answers
62
views
Can the EM algorithm be used to perform model selection?
Suppose that the samples $\mathcal{D} = \{x_1,x_2,\dots,x_N\}$ were independently sampled from a Gaussian distribution with mean $\mu$ and variance $\sigma^2$. However, suppose that we did not know ...
- 3,179
0
votes
0
answers
12
views
Residual Diagnostics of an Approximate Dynamic Factor Model estimated through EM-Algorithm
I am estimating and forecasting macroeconomic variables using an approximate dynamic factor model using EM-Algorithm. The model allows the idiosyncratic terms to follow an AR(1) process.
So in OLS ...
- 109
1
vote
1
answer
229
views
Derivation of the M-step in EM algorithm for a three-dimensional panel mixture model
I have a question regarding the estimation of a latent-class gaussian mixture model, where the model is for three dimensional panel data set with individuals $i$, in country $j$ in time $t$.
I want ...
- 21
0
votes
0
answers
39
views
Applying EM Algorithm to 3x3 Contingency Table
Given the contingency table
$$
\begin{array}{l}
\begin{array}{l|ccc}
& \text { Alcholoic } & \text {Non-Alcholic} & \text { Not Answered } \\
\hline \text { Male } & n_{11} & n_{...
- 1
3
votes
2
answers
39
views
Practical considerations for EM clustering
EM algorithm guarantees finding a local rather then global minimum of the likelihood. As a consequence, the results are dependent on the initial conditions (e.g., if randomly choosing the initial ...
- 1,800
2
votes
1
answer
686
views
How can I derive the EM algorithm for a mixture of two Bernoulli distributions?
How can I derive the E-step and M-step in the EM algorithm for a mixture of two Bernoulli distributions? Note that I am aware that there are several notes online that explain how to do this for the ...
- 3,179
15
votes
2
answers
674
views
Finding category with maximum likelihood method
Let's say that we had an information for men and women heights.
R code:
...
- 153
5
votes
1
answer
178
views
Why use the EM Algorithm and not just marginalise the complete likelihood?
On the wikipedia article for Expectation-Maximization it states
Given the statistical model which generates a set $\mathbf{X}$ of observed data, a set of unobserved latent data or missing values $\...
- 91
0
votes
0
answers
18
views
EM (?) algorithm with dependent observations but the prior seems not informative
How can we derive the (EM?) algorithm to estimate $b$ if we suppose
$G_j = bg_j$ for all $j$,
the priors (?) of the observations $\hat{G}_j \sim N(\hat{G}_j | G_j, (s_{G, j})^2)$, $\hat{g}_j \sim N(\...
- 101
0
votes
1
answer
41
views
EM algorithm when there are too many components to calculate the function Q
Assuming a regression model as follow:
$$\mathbf{y} = \mathbf{x}\beta + \mathbf{\varepsilon}$$
where
$\mathbf{y}=(y_1,...,y_n)^T\in\mathbb{R}^{n\times 1}$,
$\mathbf{x}=(x_1,...,x_n)^T\in\mathbb{R}^{n\...
- 11
1
vote
2
answers
123
views
Is this interpretation of the EM algorithm correct?
Suppose we want to estimate the parameter $\theta$ of the distribution $p(z;\theta)$ of a random variable $Z$. If we had the i.i.d samples $z_1, z_2, \dots, z_N$, we could easily do this using maximum ...
- 3,179
0
votes
0
answers
37
views
Maximum likelihood expectation maximization (for image segmentation)
I am trying to wrap my head around the concept of using Maximum likelihood expectation maximization for image segmentation. I understand the concept of maximum likelihood as a way of finding the most ...
- 13
1
vote
1
answer
110
views
EM algorithm and random effects
I am trying to learn how to fit random effects/mixed effects models. I read on this article that quasi-likelihood, numerical integration and MCMC are the most common methods.
My question is:
can we ...
- 2,565
0
votes
0
answers
33
views
Reference Request: Variational Expectation-Maximization algorithm for Latent Dirichlet Allocation with an added time component
This link has a pretty good runthrough on the variational inference (via variational E-M) for LDA with calculations expanded and explained.
I am now considering a modified LDA which adds a time ...
- 1
5
votes
1
answer
661
views
K Means as a special case of GMM (using EM Algorithm)
I am looking for a tutorial/gentle introduction (preferably with mathematics/proofs) on K-means as a special case of Gaussian Mixture Model using the EM Algorithm.
I have found this: https://www....
- 8,165
1
vote
1
answer
128
views
Understanding numerical example of expectation maximization
I was trying to understand Expectation maximization algorithm. This is how it is defined in Andrew Ng's Stanford CS229 course:
$$
\text{Repeat until convergence \{}\quad\quad\quad\quad\quad\quad\...
- 123
0
votes
0
answers
28
views
Expected 0-1 under multinomial distribution
Suppose the world generates a single observation $x \sim \text{multinomial}(\theta)$, where the parameter vector $\theta = (\theta_1,\ldots, \theta_k)$ with $\theta_i\geq 0$ and $\theta_1+\ldots+\...
- 121