# Questions tagged [expectation-maximization]

An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.

505 questions
Filter by
Sorted by
Tagged with
42 views

### EM Algorithm For Bipolar Normal Distribution

Question: Let $x_1, \dots, x_m$ be an i.i.d. sample from a normal density with mean $\mu$ and variance $\sigma^2$. Suppose for each $x_i$ we observe $y_i = |x_i|$ . Formulate an EM algorithm for ...
41 views

### Nonparametric mixture estimation

Let's assume that we have two samples $\{X_i\}_{i=1..N}$ and $\{Y_i\}_{i=1..M}$ corresponding to random variables $X$ and $Y$. Let there also be a sample $\{Z_i\}_{i=1..K}$ corresponding to random ...
42 views

46 views

### M step EM algorithm in Mixture Models. Expected value of the indicator variable under the posterior [closed]

I am not able to solve the following expectation. In the EM algorithm, the first step in the M step is to compute the expected value of $\log p(x,z)$ where $x$ are observations and $z$ indicator ...
60 views

### Why marginal likelihood is optimized in expectation maximization?

Suppose we would like maximize a likelihood function $p(\mathbf x, \mathbf z| \theta)$, where $\mathbf x$ is observed, $\mathbf z$ is a latent variable, and $\theta$ is the collection of model ...
36 views

### MCMC inside Expectation Maximization

I wish to optimize the following likelihood function for parameter $\Theta$: $$p(D|\Theta)=\int_X\int_Y p(x, y, D|\Theta)dydx$$ where $X$ and $Y$ are latent variables and only $D$ is observed. I ...
69 views

### Exponential Reestimation Formula in EM Algorithm

I'm trying to understand how to reestimate parameters, as part of the EM algorithm. As a simple example, I'm trying to derive the reestimation formula for an exponential distribution. Here's the setup:...
76 views

### How to fit a mixture of a normal and a half normal distribution?

I tried Expectation-Maximization (EM) based fitting using the mixfit function from the mixR package in the R environment. It yielded a normal mixture model with 2 components: 1) pi 0.21, mu: 0.47, sd: ...
26 views

### How to estimate parameters of a hierarchical model?

I have the following hierarchical model, where $t$ stands for time and $y_t, x_{1t}, x_{2t}, \dots, z_{1t}, z_{2t}, \dots$ are known. I want to estimate the parameters of the model using a ...
14 views

### Compute membership probabilities in E-step of EM algorithm with log-densities instead of densities

As an exercise I have implemented the EM algorithm for Gaussian mixtures, however, I have the problem that in high dimensions the densities of data points become so small that I get a numerical ...
25 views

### Clarification of a passage of Bishop about EM algorithm

I am trying to get a very good grasp of the EM algorithm, including the MCEM variant. There is one little passage of the famous Bishop book (pag 440, Mixture Models and EM) where it says: Now ...
45 views

### Derivation of EM in bishop

I'm working through chapter 9 in Bishop (Mixture models and EM) and I'm stuck on equation 9.29. For those without the book: Bishop states that the log likelihood for a latent variable model with ...
30 views

### Random forest (or regression) when predictors are intervals

I'm trying to fit a Random Forest model where the predictors are reported as an interval rather than a point estimate. The structure of each data point is a triplet $(\bar{y_i}, x_{i1}, x_{i2})$ where ...
24 views

### Evaluating goodness of fit of a model estimated with EM-algorithm (with AIC or BIC)

I am learning a Hidden Markov Model with time varying transition probabilities depending on different features. I do this by estimating the model parameters with the EM-algorithm. Now I would like to ...
54 views

### E-step of E-M algorithm with missing data

I am learning expectation-maximization (E-M) algorithm on Coursera and during the course the teacher says that it can be used to handle missing data when fitting Gaussian mixtures (GM) but did not ...
101 views

### EM algorithm for mixture of categorical distributions instantly stabilizes

Brief Summary of Question I'm trying to fit a mixture model of categorical distributions (see https://en.wikipedia.org/wiki/Categorical_distribution). The expectation at the second time step is ...
56 views

### Use Expectation-Maximization algorithm for obtaining maximal likelihood estimator

For $X = {(Z_{i}, Y_{i}) : i = 1, ... ,n}$, consider the model: $Y_{i} = \beta_{1} + \beta_{2}Z_{i} + \epsilon_{i}$ where $\epsilon_{1}, ... ,\epsilon_{n}$ are i.i.d $N(0,\sigma^2)$, $Z_{i},...Z_{i}$ ...
14 views

### Applying the EM algorithm for model with 'fixed' coefficients

My problem is that I want to apply the EM algorithm on a stochastic model (knowing full well that the model is misspecified). \begin{align} log(y_{t}^2) &= h_t + log(\epsilon_t ^2) \\ h_{t+1} = \...
158 views

### Derivation of maximum likelihood for a Gaussian mixture model

I'm working my way through the derivation of EM in Bishop (p. 435). I'm stuck trying to derive to MLE for $\mu_k$ for the gaussian mixture model. Basically I get an extra sum in the numerator. For ...
16 views

### Bayesian PCA, problem understanding Expectation Maximization scheme

I'm reading the following article https://papers.nips.cc/paper/1549-bayesian-pca.pdf of Christophe M. Bishop. I've understood the general method, however, I have trouble understanding the EM scheme ...
46 views

### Why don't we treat the mean and variances in EM algorithm as latent variables

I know how the Expectation Maximization works. What I fail to understand is why only the mixture components are treated as latent variables and why not the mean and variances values of the K gaussians?...
28 views

### What is the Q distribution in expectation maximization in the following explanation?

I am reading a blog on expectation maximization - http://krasserm.github.io/2019/11/21/latent-variable-models-part-1/ Here, I encounter the following expression: When you look at the above ...
19 views

### In expectation maximization, why do we have a latent variable distribution for every sample of the data

I am reading this blog on expectation maximization - http://krasserm.github.io/2019/11/21/latent-variable-models-part-1/ Starting the section where the author starts explaining how EM is done in the ...
100 views

### Does EM algorithm require us to know the joint (predictive) distribution of the latent variables $Z$ when $Z$ is two-dimensional?

In its general form the E-step of the EM algorithm finds the expectation $$Q(\theta|\theta') =\int \log[ p(Y,Z | \theta)] p(Z|Y,\theta') d Z$$ where $Y$ the data, $Z$ the latent variables, $\theta'$...
165 views

### ELBO maximization with SGD

In cases such as Gaussian mixture models, there's is no closed-term solution for the original likelihood maximization. Maximizing the ELBO, however, does have analytical update formulas (i.e. formulas ...
119 views

### interpretation of the estimated parameters of a gaussian mixture model

I need to find/fit a model for the color of an object. Suppose its color is generally yellow and we have 10000-by-3 data which are pixel values for R, G, B channels. Firstly I choose a Multivariate ...
50 views

### Gaussian Mixture model - Penalized log-likelihood in EM algorithm not monotone increasing

I am working on a multivariate Gaussian Mixture Model in R. The goal is to do regularized clustering on the data, where each component represents a cluster. I wrote an EM algorithm to maximize a ...