Questions tagged [expectation-maximization]
An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.
505
questions
2
votes
1answer
42 views
EM Algorithm For Bipolar Normal Distribution
Question:
Let $x_1, \dots, x_m$ be an i.i.d. sample from a normal density with mean $\mu$ and variance $\sigma^2$.
Suppose for each $x_i$ we observe $y_i = |x_i|$ . Formulate an EM algorithm for
...
2
votes
1answer
41 views
Nonparametric mixture estimation
Let's assume that we have two samples $\{X_i\}_{i=1..N}$ and $\{Y_i\}_{i=1..M}$ corresponding to random variables $X$ and $Y$. Let there also be a sample $\{Z_i\}_{i=1..K}$ corresponding to random ...
0
votes
0answers
42 views
Calculating ELBO in EM algorithm
In Andrew Ng's CS229 notes, a nice derivation of EM algorithm is given. With some minor notation modifications, the algorithm is written as follows:
E-step: For each i, $Q_i(z_i)=p(z_i|x_i, \theta^{...
5
votes
1answer
36 views
Finding the Q function for the EM algorithm
I have a situation where $X_1,...X_n$ come from $N(\mu,1)$ and there is a realization of 10 $x$ values. I want to use the EM algorithm to work out the MLE.
So, I am trying to compute the expected ...
2
votes
1answer
33 views
Why maximizing the expected value of log likelihood under the posterior distribution of latent variables maximize the observed data log-likelihood?
I am trying to understand the Expectation-Maximization algorithm and I am not able to get the intuition of a particular step. I am able to verify the mathematical derivation but I want to understand ...
0
votes
0answers
19 views
Fixing the parameters of the variational distribution in Expectation-Maximization
Consider directed graphical model $z \to x$ (with $z$ unobserved and $x$ observed). The evidence lower bound on the log-likelihood $\log p(x) = \log \sum_z p(x, z; \theta)$ for parameters $\theta$ (...
1
vote
1answer
28 views
How to get number of iterations in EM-algorithm using R mclust gaussian mixture model
I am clustering data using the mclust function from the R mclust package. I am struggling to get the number of iterations the EM ...
1
vote
1answer
23 views
derivation of E step in EM algorithm for pLSA via Lagrangian
I have trouble deriving the EM algorithm for the Probabilistic latent semantic analysis (pLSA) model via Lagrange multipliers.
I model the missing data $Q_{zij} \in \{0,1\}$ for word $w_j$ in document ...
1
vote
2answers
40 views
Understanding Bishop on EM for HMM's
I'm reading page 616 of Bishop's PRML (pdf), which introduces EM for hidden markov models with categorical hidden states and arbitrary emission distributions. Bishop defines $z_n$ as the hidden state ...
0
votes
0answers
6 views
EM, MCMC and VB, What are their similarity, difference and use scenorio?
I am recently learning MCMC, and realized some connections between MCMC, EM and VB (variational Bayesian). However, I did not find any discussion that can clear my mind on their similarity, difference ...
1
vote
1answer
23 views
Are EM and MCMC closed-form analytical solution
I came across the concept of closed-form analytical solution while learning MCMC and got a bit confused. At this point, my understanding is that the problem does not have closed-form solution if it ...
0
votes
0answers
15 views
The performance of EM algorithm at different local maximas
Setup: Say I have a problem (any problem that can be modeled by a probabilistic model) to solve, this problem involves some unknown parameters to estimate. I model this problem in a probabilistic ...
3
votes
2answers
99 views
What is the relationship between VAE and EM algorithm?
What's the relationship between Variational Autoencoders and the Expectation Maximization Algorithm?
I know that the EM algorithm is used in latent variable models, specifically to do maximum ...
0
votes
0answers
18 views
Solving Markov Switching Autoregressiv (MS-AR) model
I am currently trying to estimate a MS-AR model of the form
$$y_t - \mu_{s_t} = \sum_{l=1}^{p}\phi_l(y_{t-l}-\mu_{s_{t-l}}) + \epsilon_t$$
with $p=4$ and $n=3$ regimes. Further, the variance terms are ...
2
votes
1answer
115 views
EM Algorithm Derivation, Discrete Case
Just wanted to ask whether the following derivation is correct:
Suppose $X$ is a vector of observed random variables, $Z$ is a vector of unobserved random variables and $\theta$ is a vector of ...
2
votes
0answers
16 views
Multi-layer regression (with non-linearity in between): Iterated regression algorithm?
Suppose we observe $z_1,\dots,z_n\in\mathbb{R}$ and $x_1,\dots,x_n\in\mathbb{R}^b$ generated from the following model, for $i=1,\dots,n$:
$$y_i=x_i^\top \beta,$$
$$z_i={f(y_i)}^\top \gamma + \...
0
votes
0answers
13 views
What assumption need to hold for Monte Carlo EM algorithm?
I have a hierarchical(/multilevel) model from which I want to estimate its parameters. I do by the use of Maximimum Likelihood Estimation via the (Markov Chain) Monte Carlo EM algorithm. When solving ...
4
votes
1answer
182 views
Derivation of M step for Gaussian mixture model
Summary
So to summarize my question, how can I take
\begin{align}
= \sum_{i=1}^{n}W_{i1} \left(log (1-\sum_{j=2}^{K}\pi_j) -\frac{1}{2} log(|\Sigma_1|) -\frac{d}{2} log(2\pi) -\frac{1}{2}(x_i-\mu_1)^{...
0
votes
0answers
16 views
Gaussian mixture model parameter updates derivation?
I've been following a helpful tutorial and I'm trying to understand the parameter updates.
For example, the mu_k parameter update is below. I'm unsure why the sum(Bk) does not cancel out as it's in ...
3
votes
0answers
31 views
Expectation-Maximisation derivations [duplicate]
I've come across a few different sources on expectation-maximisation which I can't quite match up.
The CS229 lecture 8 [1] states that the function we must write down and maximise is:
$$
Q_1 = \sum_{...
0
votes
0answers
11 views
Is this the correct expression for variational distribution q(t) for expectation maximization algorithm?
Just hoping for a sanity check on my equations below.
Ultimate goal: apply the EM algorithm to a GMM problem. First I need to come up with an expression for
the variational distribution q(t) where ...
1
vote
1answer
26 views
The connection between the expectation in expectation maximization and the importance sampling?
The log-likelihood of the EM algorithm can be expressed as
\begin{align}
\ell(\theta, x) &= \log p(x|\theta) \\
&= \log \sum_z p(x, z|\theta) \\
&= \log \sum_z \frac{q(z|x)}{q(z|x)}p(x,z|...
0
votes
0answers
12 views
Fitting a mixture model distribution to kurtotic data
I need to fit a parametric distribution to data that has non-zero (unknown) kurtosis. First I tried to fit a Pearson type VII / Student's t, but the fitting is especially poor in the two tails, ...
0
votes
1answer
19 views
Nonlinear filtering with unknown model parameters
I have a Markov chain with scalar states $x_t$ evolving according to
$$x_t = x_{t-1} + \boldsymbol\vartheta^{\rm T} {\boldsymbol\varphi_{t-1}}(x_{t-1}) + w_t$$
with unknown but constant parameters $\...
0
votes
1answer
46 views
M step EM algorithm in Mixture Models. Expected value of the indicator variable under the posterior [closed]
I am not able to solve the following expectation. In the EM algorithm, the first step in the M step is to compute the expected value of $\log p(x,z)$ where $x$ are observations and $z$ indicator ...
1
vote
1answer
60 views
Why marginal likelihood is optimized in expectation maximization?
Suppose we would like maximize a likelihood function $p(\mathbf x, \mathbf z| \theta)$, where $\mathbf x$ is observed, $\mathbf z$ is a latent variable, and $\theta$ is the collection of model ...
1
vote
1answer
36 views
MCMC inside Expectation Maximization
I wish to optimize the following likelihood function for parameter $\Theta$:
$$p(D|\Theta)=\int_X\int_Y p(x, y, D|\Theta)dydx$$
where $X$ and $Y$ are latent variables and only $D$ is observed. I ...
2
votes
1answer
69 views
Exponential Reestimation Formula in EM Algorithm
I'm trying to understand how to reestimate parameters, as part of the EM algorithm. As a simple example, I'm trying to derive the reestimation formula for an exponential distribution. Here's the setup:...
-1
votes
1answer
76 views
How to fit a mixture of a normal and a half normal distribution?
I tried Expectation-Maximization (EM) based fitting using the mixfit function from the mixR package in the R environment. It yielded a normal mixture model with 2 components:
1) pi 0.21, mu: 0.47, sd: ...
0
votes
0answers
26 views
How to estimate parameters of a hierarchical model?
I have the following hierarchical model,
where $t$ stands for time and $y_t, x_{1t}, x_{2t}, \dots, z_{1t}, z_{2t}, \dots$ are known. I want to estimate the parameters of the model using a ...
0
votes
0answers
14 views
Compute membership probabilities in E-step of EM algorithm with log-densities instead of densities
As an exercise I have implemented the EM algorithm for Gaussian mixtures, however, I have the problem that in high dimensions the densities of data points become so small that I get a numerical ...
0
votes
0answers
25 views
Clarification of a passage of Bishop about EM algorithm
I am trying to get a very good grasp of the EM algorithm, including the MCEM variant. There is one little passage of the famous Bishop book (pag 440, Mixture Models and EM) where it says:
Now ...
2
votes
1answer
45 views
Derivation of EM in bishop
I'm working through chapter 9 in Bishop (Mixture models and EM) and I'm stuck on equation 9.29. For those without the book:
Bishop states that the log likelihood for a latent variable model with
...
1
vote
0answers
30 views
Random forest (or regression) when predictors are intervals
I'm trying to fit a Random Forest model where the predictors are reported as an interval rather than a point estimate. The structure of each data point is a triplet $(\bar{y_i}, x_{i1}, x_{i2})$ where ...
0
votes
0answers
24 views
Evaluating goodness of fit of a model estimated with EM-algorithm (with AIC or BIC)
I am learning a Hidden Markov Model with time varying transition probabilities depending on different features. I do this by estimating the model parameters with the EM-algorithm. Now I would like to ...
0
votes
0answers
54 views
E-step of E-M algorithm with missing data
I am learning expectation-maximization (E-M) algorithm on Coursera and during the course the teacher says that it can be used to handle missing data when fitting Gaussian mixtures (GM) but did not ...
0
votes
0answers
101 views
EM algorithm for mixture of categorical distributions instantly stabilizes
Brief Summary of Question
I'm trying to fit a mixture model of categorical distributions (see https://en.wikipedia.org/wiki/Categorical_distribution). The expectation at the second time step is ...
1
vote
1answer
56 views
Use Expectation-Maximization algorithm for obtaining maximal likelihood estimator
For $X = {(Z_{i}, Y_{i}) : i = 1, ... ,n}$, consider the model:
$Y_{i} = \beta_{1} + \beta_{2}Z_{i} + \epsilon_{i}$
where $\epsilon_{1}, ... ,\epsilon_{n}$ are i.i.d $N(0,\sigma^2)$, $Z_{i},...Z_{i}$ ...
0
votes
0answers
14 views
Applying the EM algorithm for model with 'fixed' coefficients
My problem is that I want to apply the EM algorithm on a stochastic model (knowing full well that the model is misspecified).
\begin{align} log(y_{t}^2) &= h_t + log(\epsilon_t ^2) \\
h_{t+1} = \...
4
votes
1answer
158 views
Derivation of maximum likelihood for a Gaussian mixture model
I'm working my way through the derivation of EM in Bishop (p. 435).
I'm stuck trying to derive to MLE for $\mu_k$ for the gaussian mixture model.
Basically I get an extra sum in the numerator.
For ...
0
votes
0answers
16 views
Bayesian PCA, problem understanding Expectation Maximization scheme
I'm reading the following article https://papers.nips.cc/paper/1549-bayesian-pca.pdf of Christophe M. Bishop. I've understood the general method, however, I have trouble understanding the EM scheme ...
2
votes
1answer
46 views
Why don't we treat the mean and variances in EM algorithm as latent variables
I know how the Expectation Maximization works. What I fail to understand is why only the mixture components are treated as latent variables and why not the mean and variances values of the K gaussians?...
0
votes
1answer
28 views
What is the Q distribution in expectation maximization in the following explanation?
I am reading a blog on expectation maximization - http://krasserm.github.io/2019/11/21/latent-variable-models-part-1/
Here, I encounter the following expression:
When you look at the above ...
0
votes
1answer
19 views
In expectation maximization, why do we have a latent variable distribution for every sample of the data
I am reading this blog on expectation maximization - http://krasserm.github.io/2019/11/21/latent-variable-models-part-1/
Starting the section where the author starts explaining how EM is done in the ...
5
votes
0answers
100 views
Does EM algorithm require us to know the joint (predictive) distribution of the latent variables $Z$ when $Z$ is two-dimensional?
In its general form the E-step of the EM algorithm finds the expectation
$$ Q(\theta|\theta') =\int \log[ p(Y,Z | \theta)] p(Z|Y,\theta') d Z$$
where $Y$ the data, $Z$ the latent variables, $\theta'$...
4
votes
1answer
165 views
ELBO maximization with SGD
In cases such as Gaussian mixture models, there's is no closed-term solution for the original likelihood maximization. Maximizing the ELBO, however, does have analytical update formulas (i.e. formulas ...
2
votes
2answers
119 views
interpretation of the estimated parameters of a gaussian mixture model
I need to find/fit a model for the color of an object. Suppose its color is generally yellow and we have 10000-by-3 data which are pixel values for R, G, B channels.
Firstly I choose a Multivariate ...
1
vote
1answer
50 views
Gaussian Mixture model - Penalized log-likelihood in EM algorithm not monotone increasing
I am working on a multivariate Gaussian Mixture Model in R. The goal is to do regularized clustering on the data, where each component represents a cluster.
I wrote an EM algorithm to maximize a ...
0
votes
0answers
24 views
How to calculate weights in EM-algorithm?
Assume that we have 2 clusters with some initial model:
Can you please explain how weights are calculated?
1
vote
1answer
21 views
Fitting mixture model on data with duplicate values
What is the correct procedure to fit finite mixture models on data with many duplicate values using EM? Let's say I have N(0,1) distributed data and try to fit a 2 component mixture using EM. There ...
