Questions tagged [expectation-maximization]

An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.

Filter by
Sorted by
Tagged with
0
votes
1answer
13 views

In expectation maximization, why do we have a latent variable distribution for every sample of the data

I am reading this blog on expectation maximization - http://krasserm.github.io/2019/11/21/latent-variable-models-part-1/ Starting the section where the author starts explaining how EM is done in the ...
2
votes
0answers
69 views
+250

Does EM algorithm require us to know the joint (predictive) distribution of the latent variables $Z$ when $Z$ is two-dimensional?

In its general form the E-step of the EM algorithm finds the expectation $$ Q(\theta|\theta') =\int \log[ p(Y,Z | \theta)] p(Z|Y,\theta') d Z$$ where $Y$ the data, $Z$ the latent variables, $\theta'$...
4
votes
1answer
131 views

ELBO maximization with SGD

In cases such as Gaussian mixture models, there's is no closed-term solution for the original likelihood maximization. Maximizing the ELBO, however, does have analytical update formulas (i.e. formulas ...
0
votes
0answers
29 views

Compound distribution

I'm trying to compute a maximum likelihood of compound Poisson exponential distribution in R by using EM-algorithm method. The distribution is defined by $∑N_j=1 Y_j$ where $Y_n$ is i.i.d sequence ...
2
votes
2answers
65 views

interpretation of the estimated parameters of a gaussian mixture model

I need to find/fit a model for the color of an object. Suppose its color is generally yellow and we have 10000-by-3 data which are pixel values for R, G, B channels. Firstly I choose a Multivariate ...
1
vote
1answer
17 views

Gaussian Mixture model - Penalized log-likelihood in EM algorithm not monotone increasing

I am working on a multivariate Gaussian Mixture Model in R. The goal is to do regularized clustering on the data, where each component represents a cluster. I wrote an EM algorithm to maximize a ...
0
votes
0answers
17 views

How to calculate weights in EM-algorithm?

Assume that we have 2 clusters with some initial model: Can you please explain how weights are calculated?
2
votes
3answers
533 views

GMM EM algorithm complexity per iteration

I was fitting GMM clusters with diagonal covariance on my data using EM with $n$ (=5e6) points, each having $m$ (=160) ...
1
vote
1answer
219 views

The ways to normalize the likelihood in EM algorithm

In Wikepedia it states that: In the simplest cases, normalization of ratings means adjusting values measured on different scales to a notionally common scale, often prior to averaging. And ...
0
votes
0answers
31 views

assessing the stability of importance (sampling) weights

I have read that when importance weights are used, the stability (variability) of the weights should be assessed (Levine and Casella, 2001) -- however, I wonder how this might be accomplished. For ...
1
vote
1answer
11 views

Fitting mixture model on data with duplicate values

What is the correct procedure to fit finite mixture models on data with many duplicate values using EM? Let's say I have N(0,1) distributed data and try to fit a 2 component mixture using EM. There ...
1
vote
0answers
20 views

Confusion in Sampling using the IP algorithm (Bishop PRML)

I'm reading Bishop's PRML p. 537 and I don't understand one piece of the IP (data augmentation) algorithm. Namely, the part that says "we use the samples $\{\mathbf{Z}^{(l)}\}$ obtained in the I step ...
1
vote
0answers
26 views

Expectation Maximisation vs Expectation Propagation in the context of Bayesian Networks

I am confused about Expectation Maximisation and Expectation Propagation algorithms in the context of Bayesian Networks, especially whether one comprise another. What is the difference between ...
0
votes
0answers
14 views

Newton-Raphson method to solve for dof when performing MLE of a multivariate Student-t distribution using EM

I am reading the derivation of EM algorithm to estimate the maximum likelihood of a multivariate Student-t distribution $\mathcal{T}(\mathbf{x} \vert \pmb{\mu}, \pmb{\Sigma}, \nu)$ in Kevin Murphy's ...
0
votes
1answer
243 views

EM Derivation for Dawid-Skene Model

I am trying to derive the EM update equations for the Dawid-Skene model. Following the notation in Bayesian Classifier Combination by Kim and Ghahramani, $i$ is the index of the data point, $t_i$ is ...
0
votes
0answers
42 views

Why Expectation and Maximization algorithm not used in Machine Learning while Gradient Descent algorithm used in Machine Learning?

I know that Newton Raphson, Expectation & Maximization, and Gradient Descent are all known to be optimization methods. Somehow, I wonder why Gradient Descent is chosen to be used in most of ...
0
votes
0answers
18 views

Do I impute missing values with the response?

I have a dataset with missing values in both predictors and the response. As far as I know, the data are missing not at random, so I cannot simply use listwise deletion. Instead, I employed the EM ...
3
votes
2answers
4k views

Simple Explanation of Baum Welch/Viterbi

I'm looking for a very simple explanation as possible for Baum Welch and Viterbi for HMMs with a straightforward well annotated example. Almost all of the explanations I find on the net invariably ...
2
votes
0answers
18 views

Can someone verify if the following Bayesian Information Criterion (BIC) model selection algorithm is correct for Gaussian mixture models?

I am trying to find an automated way of picking the number of clusters $K \in \mathbb{N}$ for unsupervised learning scenarios, specifically for GMM. I was suggested to use something called the "...
4
votes
0answers
39 views

Expectation Maximisation (EM) Algorithm

Some of my parameters do not have a closed form solution. Thus, for these parameters the M-step is implemented via a one-step Newton-Raphson update, i.e., \begin{equation} \theta^{t+1} = \theta^t - \...
0
votes
0answers
12 views

Outlier detection with EM

I am interested in using expectation maximization for outlier detection. In the literature this is usually done assuming that the data of interest are normally distributed while the outliers are ...
0
votes
0answers
12 views

Using expectation maximization for robust regression

What are the advantages/disadvantages of using EM for robust estimation vs. the robust estimation with Huber or Tukey loss functions?
2
votes
2answers
49 views

EM-algorithm for two clusters (when one of the distributions is uniform)

I am having a hard time with the EM-algorithm. Here's the problem that I am trying to solve. Dealing with noisy annotations is a common problem in computer vision, especially when using ...
3
votes
2answers
77 views

Accounting for uncertain information (few observations) in a prior (empirial Bayes)

I did not really know how to choose an adequate title for this question, so please feel free to change it. I have a weird case wherein frequentist and Bayesian philosophies come together. I am ...
1
vote
0answers
8 views

Consistency of EM for missing data in non-parametric setting

When we have missing data, a parametric model, and an expectation-maximization procedure, and we want to show that our procedure leads to consistent estimators, we can sometimes set up score functions ...
1
vote
1answer
38 views

Calculation of AIC in finite mixture modeling

I have a question about calculation the AIC to find my optimal amount of clusters. I am applying mixture modeling with the EM algorithm. I know the formula AIC = -2ln(log-lik) + 2k. These are my log-...
0
votes
0answers
13 views

Analysing faithful dataset in R using GMM

I have got a project on analysing the faithful data in R found in the package "datasets" and called using data(faithful) which is the data set off eruption time and waiting time of the Old Faithful ...
1
vote
1answer
187 views

Regularizing soft kmeans with entropy

So in classical fuzzy k-means clustering, the objective function is $\sum_i \sum_j u_{ij} \|x_i - c_j\|^2$ Now, we want to regularize this objective function using the entropy: $\sum_i^n H(U_i) = - \...
1
vote
0answers
29 views

Are my data a good candidate for EM imputation followed by exploratory factor analysis?

I am doing Exploratory Factor Analysis (EFA) in R, using principal axis factoring in the psych package. I have missing data that prevent me getting factor scores, so I am imputing data. I am using ...
0
votes
0answers
19 views

Cosine Similarity for Classification to EM Cluster?

Perhaps my question sounds naive, uncovering the very little knowledge that I have in the field of Statistics, but is very urgent to get a solid answer or trigger for further insights for my concerns. ...
1
vote
1answer
134 views

EM algorithm with dependent observations

I am trying to implement an EM algorithm for dependent observations. Specifically, I am dealing with families where the hidden variables $Z$ of the children are dependent on the hidden variables of ...
1
vote
1answer
2k views

Understanding the log-likelihood (score) in scikit-learn GMM

I have been training a GMM (Gaussian Mixture, clustering / unsupervised) on two version of the same dataset: one training with all its features and one training after a PCA truncated to its 2 first ...
2
votes
1answer
437 views

Gaussian clusters and original distributions

In Gaussian clustering (i.e. General Mixture Models) we model the data with some clusters. For example, in the below figure, we have two clusters $C_1, C_2$, each of which are modeled with a Gaussian (...
6
votes
1answer
3k views

Relation between MAP, EM, and MLE

I am a beginner in machine learning. I can do programming fine but the theory confuses me a lot of the times. What is the relation between Maximum Likelihood Estimation (MLE), Maximum A posteriori (...
4
votes
1answer
293 views

How to make sense of the EM algorithm expressed in terms of Kullback-Leibler divergence?

In the textbook All of Statistics by Wasserman, the algorithm is expressed as: Pick a starting value $\theta^0$. (E-Step). Calculate: $$ J(\theta|\theta^j) = E_{\theta^j} \left(\log \dfrac{f(Y^n, Z^...
2
votes
1answer
233 views

Another EM-algorithm problem

I have the following problem: I have a random vector $y$ which has length $l$. The first $z$ bits come from a Bernoulli random variable with parameter $\theta_1$ and the next $l-z$ come from a ...
1
vote
1answer
78 views

Using AIC/BIC within cross-validation for likelihood based loss functions

For a course I am teaching, I am having my students fit a Gaussian mixture model using MLEs via the EM algorithm to a bivariate dataset. I have asked the students to use use cross-validation to choose ...
1
vote
0answers
39 views

A trivial question about EM algorithm theory

In "The EM Algorithm and Extensions", second edition, from Geoffrey J. McLachlan and Thriyambakam Krishnan, X is the latent variable, and Y is de observed (incomplete) variable I'm little confuse ...
4
votes
1answer
428 views

Expectation Maximization intuitive explanation

Given a set of events {A, B, C, D, E} that occur once each month for n years: ...
1
vote
0answers
26 views

EM algorithm, Elements of Statistical Learning, expectation of log-likelihood

While I am reading ESL, I have some questions in chapter 8 (Model inference and averaging). Specifically,8.5.2 The EM algorithm in general. This part explains how EM works, in general, referring to ...
6
votes
1answer
148 views

Does marginalization of some of the latent variables improve convergence in EM?

Given a likelihood to maximize $$ \log p(x | \theta) $$ Imagine that, in order to apply EM, we can augment the model with one or two latent variables. In that case, we can derive two lower bounds: $...
4
votes
2answers
787 views

What is a “tempered EM algorithm”?

In the paper of Probabilistic Latent Semantic Analysis by Hofmann, the author fits the model for document $\times$ word matrix through EM Algorithm in section 3. I was able to follow the derivation ...
0
votes
0answers
74 views

Expectation Maximization (EM) stopping criterion

In several EM description (e.g., Theory and Use of the EM Algorithm By Maya R. Gupta and Yihua Chen) I read that two tipical stopping criterions for EM are defined on the difference between log-...
18
votes
2answers
4k views

Why is optimizing a mixture of Gaussian directly computationally hard?

Consider the log likelihood of a mixture of Gaussians: $$l(S_n; \theta) = \sum^n_{t=1}\log f(x^{(t)}|\theta) = \sum^n_{t=1}\log\left\{\sum^k_{i=1}p_i f(x^{(t)}|\mu^{(i)}, \sigma^2_i)\right\}$$ I was ...
2
votes
1answer
15 views

Visualized Imputation of 2D Sample Space With Saturation Corresponding to Confidence?

I want to visualize a 2D sample space imputed from a 2D scatter of measurements. Something like a 2D hexbin color histogram would work if it were enhanced in the ...
1
vote
0answers
21 views

Is there a way to run constrained EM?

I would like to run an EM (expectation maximization) algorithm to estimate hidden Markov model parameters except, in my case, I have an extra constraint on my start and final states in my HMM. How do ...
2
votes
1answer
108 views

Figueiredo and Jain's Gaussian mixture EM convergence criterion

I have implemented and been playing around Figueiredo & Jain 's trainer in this paper http://www.lx.it.pt/~mtf/IEEE_TPAMI_2002.pdf for Gaussian mixture. Fig. 2 in the paper depicts the full ...
1
vote
2answers
28 views

Estimating latent mean and variance for a Gaussian

I have a latent Gaussian model with unknown parameters $\mu$ and $\sigma^2$. I can estimate these parameters using MLE and an EM-ish algorithm. However the solution is not stable; I end up in local ...
1
vote
1answer
46 views

State Space models: rewriting the Likelihood to estimate the covariance matrix

I have a State Space model $ \begin{matrix} Y_t & = & FX_t +R_{t}^{1/2}\epsilon_t \\ X_{t+1} & = & GX_{t}+Du_t \end{matrix} $ where $\epsilon_t$ and $u_{t}$ distributed ...
0
votes
0answers
14 views

Convergence of Modified Expectation-Maximisation Algorithm - interpreting language of question

We're going to consider a modified E-M algorithm and its convergence properties. To do so, we will first need to review the convergence of the standard E-M algorithm as I'll need to refer back to it. ...

1 2 3 4 5 10