Questions tagged [expectation-maximization]

An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.

Filter by
Sorted by
Tagged with
0 votes
0 answers
9 views

Hierarchical Expectation Maximization

Let us assume we have a generative model $y = g(\theta,x)$, where $\theta$ is a set of parameters, and $x$ is a set of latent variables. Given a datapoint $y$, I need to update the parameters $\theta$ ...
  • 315
1 vote
0 answers
12 views

Kinds of Expectation-Maximization [closed]

I'm starting to dig into the variational inference literature, as I need to solve a learning problem where the latent variables form an hierarchical structure (an hierarchical Gaussian generative ...
  • 315
2 votes
1 answer
38 views

Maximum likelihood for mixture of Bernoullis with known mixture proportions

Given the hierarchical model $$ \begin{align} k & \sim \text{Categorical}(\pi_1, \dots, \pi_K) \\ X \mid k & \sim \text{Bernoulli}(\theta_{k}) \end{align} $$ and an i.i.d. sample $X_1, \dots, ...
  • 3,257
1 vote
0 answers
14 views

How to get (Gaussian) mean and covariance of missing variables in the E-Step of the EM Algorithm for Missing Data

The M step for missing data imputation using MVN updates $$\mu^t=\frac{1}{N_D}\sum_n E[y_n]\\ \Sigma^t=\frac{1}{N_d}\sum_nE[y_n y_n^\top]-\mu^t(\mu^t)^\top $$ where $y_n=(x_n,z_n )$. However, the ...
  • 497
1 vote
0 answers
15 views

EM equality constraint using Jensen's Inequality

I'm trying to derive the ELBO as per CS229 Page 144, Eq 11.8. For E-step, the LHS and RHS must be equal: $$ \begin{align} f\left(\mathbb{E}_{z\sim Q}\left[ \frac{p(x, z; \theta)}{Q(z)}\right]\right) \...
0 votes
0 answers
26 views

In what situations can you use EM, but where you cannot compute the marginal likelihood?

Expectation Maximisation is used to find the parameters of a hierarchical model with some nuisance parameters, that need to be integrated out. The typical example is a Gaussian Mixture Model, where ...
0 votes
0 answers
13 views

Observation from each cluster from EM algorithm manually implemented

I am interesting in implementing the EM algorithm manually. And I did it. However, I am still do not know how to get the observation from each cluster? For example, after run the EM algorithm, the ...
  • 507
3 votes
1 answer
73 views

Need help understanding why the original proof of the EM algorithm had a flaw in it

I understand that in the original paper on the EM algorithm by Dempster, the proof of convergence in Theorem 2 had a flaw, and that a full proof wasn't given until CF Jeff Wu published it years later. ...
  • 2,315
0 votes
0 answers
24 views

Continuously running the M step while approximating the E step

Let us assume we need to maximize the likelihood $p(x|\theta)$ of an observed variable $x$ given a set of parameters $\theta$. However, this likelihood also depends on a set of hidden variables $z$, ...
  • 315
1 vote
1 answer
33 views

details of stochastic Expectation Maximization (EM) algorithm

I went into a paper, Online EM Algorithm for Latent Data Models (Olivier Cappé & Eric Moulines, 2009). I got confused by the first equation the authors wrote, the Q function: Here the authors ...
  • 11
0 votes
0 answers
20 views

Can overlap impact the EM algorithm?

I am using EM algorithm for clustering. I see that if there is an overlap, the accuracy is reduced. However, it is improvesd if no overlap exists. My question is, can overlap affect the performance of ...
  • 507
5 votes
0 answers
65 views

How does the EM algorithm work in Bayesian regression? [closed]

I have problem distinguishing between the latent variables $z_i$ and the parameters $\theta_i$ in EM algorithm. Suppose we have the hierarchical priors \begin{aligned} \beta|\tau,\omega &\sim \...
  • 189
2 votes
0 answers
45 views

E-Step of the EM algorithm on mixture

Im trying to implement the EM algorithm on mixture model: $pg(x) + (1-p)h(x)$ where the sample $\bar{x} = (x_1, \ldots, x_n)$, is independently generated from the mixture, and $g(x) = e^{-x}$ and $h(x)...
  • 119
2 votes
1 answer
60 views

EM algorithm on discrete random variable

Im trying to find the iteration formula for the maximum likelihood estimator of the parameter $\theta$ of the following discrete random variable, through the EM algorithm: $$P(Y = y ) = \sum_{i=0}^{\...
  • 119
0 votes
0 answers
14 views

In stochastic maximum likelihood and contrastive divergence, why do we sample from model distribution for partition function?

I have been reading the "Deep Learning" book from Ian Goodfellow. In a topic on Restricted Boltzmann Machines and how to train them, techniques like Stochastic Maximum Likelihood and ...
0 votes
1 answer
37 views

What is 'factored models'?

What is 'factored models'? I've seen the term 'factored models' in paper 'The Bayesian Structural EM Algorithm'. But I see it for the first time and it is hard to understand... I tried searching it ...
0 votes
0 answers
36 views

Solve the EM algorithm of the Gaussian mixture model

Considering a mixture of two normally distributed random variables $Y_1$ and $Y_2$ and considering the linear combination of the two: $$ \begin{gathered} Y_{1} \sim N\left(\mu_{1}, \sigma_{1}^{2}\...
  • 609
0 votes
0 answers
16 views

Expectation Maximization: How to perform "E" step in coin flipping example? [duplicate]

I recently read this primer that does a great job of explaining the principles of the Expectation-Maximization (EM) "algorithm." However, I'm confused about how they calculated the ...
  • 211
0 votes
0 answers
75 views

Hidden Markov Model - Baum Welch algorithm initialization

Currently I'm working on a problem where I have a multidimensional, continuous sequence of observations $X$ that model my response variable $y$ with two states $0$ and $1$. I assume that this sequence ...
  • 279
0 votes
0 answers
29 views

How to conduct EM algorithm when there are some outliers in GMM Models?

I'm just confused about the problem of adding an outlier component directly to the primary form of GMM models: Suppose that the observed data contains several outliers. The mixture model could be: $$ ...
  • 1
0 votes
1 answer
36 views

Conditional independence in EM algorithm

Let $X$, $\theta$ and $Z$ denote observed, parameter and latent nodes in a graphical model. The EM algorithm attempts to find a local maximum likelihood estimate $\theta^\ast$ for the likelihood of ...
  • 211
3 votes
1 answer
137 views

Is the likelihood for Gaussian mixture models still multimodal when Y is partially observed?

In discussing Gaussian mixture models (GMMs), https://normaldeviate.wordpress.com/2012/08/04/mixture-models-the-twilight-zone-of-statistics/ highlights the issue of Multimodality of the Likelihood. ...
  • 3,850
1 vote
1 answer
42 views

Issue with Casella&Berger derivation of EM likelihood equality

In the explanation of the EM (Expectation maximization) algorithm p.328 in the book "Statistical inference" by G. Casella and R. Berger, 2nd edition, they present the following: $\mathbf{Y} =...
0 votes
0 answers
17 views

Expectation-Maximization high missing rates and multiple variables

I know MICE can be used for imputation of multiple variables simultaneously. The expectation maximization approach (EM) can be used to impute missing data. Typically, one should only be using ...
0 votes
0 answers
42 views

Labeling unlabeled data (expectation maximization)

Say I have a database (Excel) consisting of 10k different dresses and accompanying attributes (column names) for each dress (sleeve length, color, pattern, ...). I would like to label each of these ...
0 votes
1 answer
151 views

How does expectation maximization compute missing data?

I have been searching for a simple example of how expectation-maximization (EM) computes missing data. All the examples I have found are based on multivariate normal models. I have seen that EM can be ...
1 vote
0 answers
89 views

Problems with convergence of the EM algorithm for a gaussian mixture regression

I have been implementing a EM-algorithm for a latent-class regression model, where every individual has a vector of observations. Currenly, I have the problem that the model does not converge. The log ...
0 votes
0 answers
90 views

EM derivation for mixture of t distributions

(Notation is at the bottom.) I'm currently self-studying the Expectation-Maximization algorithm and has learned to derive it for various mixture models including Gaussian mixture models, mixture of ...
  • 124
0 votes
0 answers
10 views

Model choice with expectation maximization: which likelihood?

When deciding about the number of mixture components using Akaike or bayesian information criteria, should one use the full likelihood or the likelihood marginalized over the latent variables? Both ...
  • 1,800
0 votes
0 answers
70 views

Benefits of Expectation Maximization for Mixture Models

What are the benefits of using expectation maximization for mixture models vs. direct maximization of the marginal likelihoods? Analytic maximization step In case of Gaussian mixtures the benefit is ...
  • 1,800
0 votes
1 answer
94 views

Computing a prior from two components in Naive Bayes

Given a model parameter $\theta$ that is composed of two distributions in a Naive Bayes classifier, how is $P(\theta)$ typically computed in practice? More specifically, from the article of Nigam et ...
0 votes
0 answers
84 views

Expectation maximization: does the likelihood always increase monotonically?

When working with (gaussian) mixture models, I always took it for a mathematical fact that the marginal likelihood increases with every iteration step. If it were not the case, it always meant an ...
  • 1,800
0 votes
0 answers
82 views

EM algorithm for multinomial data

Good afternoon , Assume we have a dataset modeled by multinomial data of K cluster. To apply EM algorithm parameters are : $${\theta }^{\left(0\right)}=({p}_{k}^{\left(0\right)},k=1,\mathrm{\dots },K)$...
  • 113
2 votes
0 answers
62 views

Can the EM algorithm be used to perform model selection?

Suppose that the samples $\mathcal{D} = \{x_1,x_2,\dots,x_N\}$ were independently sampled from a Gaussian distribution with mean $\mu$ and variance $\sigma^2$. However, suppose that we did not know ...
  • 3,179
0 votes
0 answers
12 views

Residual Diagnostics of an Approximate Dynamic Factor Model estimated through EM-Algorithm

I am estimating and forecasting macroeconomic variables using an approximate dynamic factor model using EM-Algorithm. The model allows the idiosyncratic terms to follow an AR(1) process. So in OLS ...
  • 109
1 vote
1 answer
229 views

Derivation of the M-step in EM algorithm for a three-dimensional panel mixture model

I have a question regarding the estimation of a latent-class gaussian mixture model, where the model is for three dimensional panel data set with individuals $i$, in country $j$ in time $t$. I want ...
0 votes
0 answers
39 views

Applying EM Algorithm to 3x3 Contingency Table

Given the contingency table $$ \begin{array}{l} \begin{array}{l|ccc} & \text { Alcholoic } & \text {Non-Alcholic} & \text { Not Answered } \\ \hline \text { Male } & n_{11} & n_{...
  • 1
3 votes
2 answers
39 views

Practical considerations for EM clustering

EM algorithm guarantees finding a local rather then global minimum of the likelihood. As a consequence, the results are dependent on the initial conditions (e.g., if randomly choosing the initial ...
  • 1,800
2 votes
1 answer
686 views

How can I derive the EM algorithm for a mixture of two Bernoulli distributions?

How can I derive the E-step and M-step in the EM algorithm for a mixture of two Bernoulli distributions? Note that I am aware that there are several notes online that explain how to do this for the ...
  • 3,179
15 votes
2 answers
674 views

Finding category with maximum likelihood method

Let's say that we had an information for men and women heights. R code: ...
5 votes
1 answer
178 views

Why use the EM Algorithm and not just marginalise the complete likelihood?

On the wikipedia article for Expectation-Maximization it states Given the statistical model which generates a set $\mathbf{X}$ of observed data, a set of unobserved latent data or missing values $\...
0 votes
0 answers
18 views

EM (?) algorithm with dependent observations but the prior seems not informative

How can we derive the (EM?) algorithm to estimate $b$ if we suppose $G_j = bg_j$ for all $j$, the priors (?) of the observations $\hat{G}_j \sim N(\hat{G}_j | G_j, (s_{G, j})^2)$, $\hat{g}_j \sim N(\...
  • 101
0 votes
1 answer
41 views

EM algorithm when there are too many components to calculate the function Q

Assuming a regression model as follow: $$\mathbf{y} = \mathbf{x}\beta + \mathbf{\varepsilon}$$ where $\mathbf{y}=(y_1,...,y_n)^T\in\mathbb{R}^{n\times 1}$, $\mathbf{x}=(x_1,...,x_n)^T\in\mathbb{R}^{n\...
  • 11
1 vote
2 answers
123 views

Is this interpretation of the EM algorithm correct?

Suppose we want to estimate the parameter $\theta$ of the distribution $p(z;\theta)$ of a random variable $Z$. If we had the i.i.d samples $z_1, z_2, \dots, z_N$, we could easily do this using maximum ...
  • 3,179
0 votes
0 answers
37 views

Maximum likelihood expectation maximization (for image segmentation)

I am trying to wrap my head around the concept of using Maximum likelihood expectation maximization for image segmentation. I understand the concept of maximum likelihood as a way of finding the most ...
1 vote
1 answer
110 views

EM algorithm and random effects

I am trying to learn how to fit random effects/mixed effects models. I read on this article that quasi-likelihood, numerical integration and MCMC are the most common methods. My question is: can we ...
  • 2,565
0 votes
0 answers
33 views

Reference Request: Variational Expectation-Maximization algorithm for Latent Dirichlet Allocation with an added time component

This link has a pretty good runthrough on the variational inference (via variational E-M) for LDA with calculations expanded and explained. I am now considering a modified LDA which adds a time ...
  • 1
5 votes
1 answer
661 views

K Means as a special case of GMM (using EM Algorithm)

I am looking for a tutorial/gentle introduction (preferably with mathematics/proofs) on K-means as a special case of Gaussian Mixture Model using the EM Algorithm. I have found this: https://www....
  • 8,165
1 vote
1 answer
128 views

Understanding numerical example of expectation maximization

I was trying to understand Expectation maximization algorithm. This is how it is defined in Andrew Ng's Stanford CS229 course: $$ \text{Repeat until convergence \{}\quad\quad\quad\quad\quad\quad\...
  • 123
0 votes
0 answers
28 views

Expected 0-1 under multinomial distribution

Suppose the world generates a single observation $x \sim \text{multinomial}(\theta)$, where the parameter vector $\theta = (\theta_1,\ldots, \theta_k)$ with $\theta_i\geq 0$ and $\theta_1+\ldots+\...
  • 121

1
2 3 4 5
12