Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [expectation-maximization]

An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.

2
votes
1answer
45 views

Simplification of an expectation

While attempting to simplify a combination of expectations, I'm stuck at a particular term whose simplification I'm unable to deduce. The term to be simplified is: $\mathbb{E}[X^{T}F^{T}FX]$ where $...
2
votes
1answer
30 views

Expectation of Sufficient Statistic

Consider $X \sim B(n,p)$ with pmf $P(X=x) = {{n}\choose{x}} p^x (1-p)^{n-x}$. The general exponential form of an exponential family distribution is $p(x|\theta) = f(x) g(\theta) e^{\phi(\theta)^T T(...
0
votes
0answers
91 views

rewriting ELBO to highlight the role of priors

I am reading this paper which rewrites ELBO. I am stuck in verifying the mathematics used for doing the rewriting. Essentially, the paper writes the KL term involved in ELBO as follows (equations 13 ...
1
vote
0answers
14 views

Stopping criteria for gaussian mixture models

As I can read from the source code of scikit-learn, the stopping criteria for the iterative algorithm of Expectation Maximization (in my case applied to fitting Gaussian mixture models) is to put a ...
0
votes
1answer
40 views

EM algorithm for zero truncated poisson

I find it very difficult to understand the E-step of EM algorithm in zero-truncated poisson example. Can someone explain me (mathematically) how exactly do we estimate the number of our "missing" zero ...
0
votes
0answers
26 views

finding the 'best' set

I came across this simple looking but puzzling question recently. There is a set of N tuples given [(a1,b1), ..., (aN, bN)], where a are real numbers and b are positive real numbers. We need to choose ...
2
votes
0answers
24 views

EM algorithm for mixture models, what happens when you don't know how many mixtures you have?

i am trying to learn something here so i tried to estimate the means of two normal distributions that I created. Here is the code: ...
0
votes
0answers
13 views

Is expectation maximization an approximation algorithm?

Is expectation maximization an approximation algorithm? Does it give the exact solution?
0
votes
0answers
11 views

Initializing structural expectation maximization for learning Bayes net structures

I am using bnlearn in R to learn Bayesian network structures. It has a structural.em method for learning with missing data that ...
0
votes
1answer
29 views

How to use log probabilities in PCA mixture EM algorithm

I'm trying to implement PCA mixtures (Tipping & Bishop 2006 Appendix C) on the Tobomovirus. I'll summarize the mathematical background and algorithm here: For a single PCA model, we assume a ...
0
votes
0answers
11 views

How far can local optimum deviate from the ground truth?

I am in the bioinformatic field and see numerous bioinformatic tools applying heuristic (for example EM) methods to find local optimum solution (for example SciClone). I know that local optimum may ...
0
votes
0answers
90 views

EM Algorithm for Bayesian Networks with missing data

Setting: learning parameters of Bayesian Network (BN) with missing data. Algorithm: Expectation-Maximization. Question: suppose I am in the M-step, and that in the complete data there are no ...
0
votes
0answers
10 views

What is the relationship between the EM-algorithm, forward-backward alrgorithm and Viterbi algorithms for Hidden Markov Model?

I know procedure of viterbi, EM-algorithm, and forward-backward independently for Hidden Markov Model. But what the relationship between them?
1
vote
1answer
32 views

log in the M-step of the EM algorithm

In the M-step of the EM algorithm, you have to maximize the expected log-likelihood of X with respect to z which is: $ \int d z P(Z \mid X, \theta^{old}) \ln P(X \mid Z, \theta)$. Why do we maximize ...
0
votes
1answer
25 views

Expectation of a discrete random variable that is case-defined from other discrete random variables

Background The question arises from the following real-life situation: I buy a newspaper at 3 dollars and sell it at 6 dollars. I know the demand for news paper is a binomial random variable with $n=...
1
vote
1answer
26 views

Reducibility between Gaussian Mixture Models and Gaussian Processes

I am studying gaussian processes and I have already discrete amount of knowledge in gaussian mixture models. I am here to undersrtand if with a gaussian process you can fit a gaussian mixture model. ...
0
votes
0answers
20 views

Epsilon from Bivariate Normal Distribution [duplicate]

I came across the following example from a book. I am given a dataset generated from a bivariate normal distribution: Among the data, there are missing values for the last 20 of x2i (but not for x1i)....
0
votes
1answer
13 views

Consistency between EM clusterings with varying starting point

I have a data set (~9 dimensions) in Weka and am running the EM clusterer with a fixed number of clusters. When changing the seed/initial point, the clusterings are very different. Is this expected? ...
0
votes
0answers
21 views

EM algorithm for factor analysis,the formula of diagonal matrix stuck

I am trying to learn the factor analysis of CS229,the relative lecture note is here:CS229 Lecture note9 I have stucked at the diagonal matrix formula which is at the page 9: What's the derivation of ...
0
votes
0answers
73 views

EM Derivation for Dawid-Skene Model

I am trying to derive the EM update equations for the Dawid-Skene model. Following the notation in Bayesian Classifier Combination by Kim and Ghahramani, $i$ is the index of the data point, $t_i$ is ...
0
votes
0answers
26 views

Linear regression of features inside a hidden Markov model?

I have an interesting little problem which I am trying to attack using HMMs. First, as usual, I am trying to do time-series segmentation/classification using a HMM. But the input to my HMM has an ...
1
vote
1answer
52 views

Estimating truth and confusion matrix from noisy observations with Expectation Maximization?

Suppose we have $m$ sources, each of which noisily observe the same set of $n$ independent events from the outcome set $\{A,B,C\}$. Each source has a confusion matrix, for example for source $i$: $$...
1
vote
0answers
71 views

how can I do online Expectation–maximization algorithm?

em algorithm is usually optimized iteratively between the expectation (construct a lower-bound) and the maximize the likelihood (optimize the lower-bound) to guarantee convergence. However, at each ...
0
votes
0answers
56 views

EM algorithm and AIC criteria

I am using EM algorithm to estimate the model parameters. EM-algorithm iterates until the loglikelihood is converged. After that, I need to compute AIC criteria. As known, AIC is a loglikelihood ...
1
vote
2answers
285 views

Comparing K-Means and Expectation Maximization on the dataset generated - When does K-Means perform better?

I was experimenting with K-Means and Gaussian Mixture Models (Expectation-Maximization) on the data set that I generated. Here is how the plot for two distributions looks like: Since this was ...
1
vote
0answers
22 views

Likelihood Construction for Censored Data

I am trying to understand the Expectation-Maximization algorithm, and was trying to read through this paper by Park and Lee. In section 2, "Likelihood Construction for Censored Data", they mention the ...
2
votes
2answers
268 views

Question about the latent variable in EM algorithm

In mixture models, Expectation maximization algorithm (EM) is a commonly used method to estimate the model parameters. Suppose that I have bivariate mixture model with two mixture components, with ...
1
vote
1answer
36 views

What is an Expectation Maximisation Algorithm for Markov chains?

I'm looking for an algorithm for Expectation Maximisation of a Markov chain. I am aware of the Baum-Welch algorithm for Hidden Markov Models, but I can't find an algorithm for Markov Models that are ...
0
votes
1answer
46 views

The ways to normalize the likelihood in EM algorithm

In Wikepedia it states that: In the simplest cases, normalization of ratings means adjusting values measured on different scales to a notionally common scale, often prior to averaging. And ...
0
votes
0answers
34 views

EM algorithm early stopping

Assume there are a set of latent variables $X$, a set of observed variables $Y$ and some parameters $\Theta$. I am using EM algorithm to compute $X$ and $\theta$. In the E step, it computes $p(X|Y,\...
2
votes
0answers
32 views

Is Expectation-Maximization the right type of analysis (& if so, how)?

I have 118 microbes that I have tested at various concentrations of a drug that is supposed to kill them (0.1875 - 15). However, these microbes have mutations (A-H) which confer resistance to the ...
1
vote
0answers
41 views

Optimal Sequence Problem

Lets says I have 100 people that like to buy item x. They ask me to send them a message every time I have x available to sell. Of the 100 people that like to buy item x: 1) Some people will pay more ...
1
vote
0answers
58 views

MLE for high dimensional $\theta$

I'm estimating a parameter $\theta$ in the context of covariance structure model given by $\Sigma(\theta)$. As an estimator, I use ML and computation is done by fmincon function in Matlab(using sqp ...
4
votes
0answers
201 views

Fisher information matrix in logistic regression

I am self-studying the basics of logistic regression. I came across this sentence: In logistic regression expected and observed information matrixes are equal I am aware that the information ...
1
vote
1answer
15 views

In the expectation step, why do we sometimes assign the data to a component (i.e. complete the data) instead of calculating the expected value?

Let $Y|X$ be a mixture distribution conditional on covariates $X$, with distribution function $Y(x; \sigma, \psi, \phi) = \alpha Y_1(x; \psi) + (1 - \alpha) Y_2(x; \phi)$, for the averaging parameter $...
0
votes
0answers
15 views

How to use the estimate given by the EM algorithm to guess at the missing value

Text: Computational Statistics 2E by Givens and Hoeting Example 4.1: Simple Exponential Density The set-up is as follows: Suppose that $Y_1, Y_2 \overset{\text{iid}}{\sim} \rm{Exp}(\theta)$ and that ...
2
votes
1answer
217 views

Gaussian Mixture Model

with the following code I fit a Gaussian Mixture Model to arbitrarily created data. The code is working. The only thing I encounter is that during the calculation of the multivariate_normal I ...
2
votes
0answers
23 views

Multivariate mixture models

I am new to mixture modeling and have successfully used bernoulli mixture models to cluster datasets of binary data. My real purpose, though is to cluster datasets with mixed data types: normal, ...
0
votes
0answers
31 views

Sample covariance matrix notation

I do not understand this notation for the sample covariance matrix (from Artificial Intelligence: A Modern Approach, Peter Norvig and Stuart J. Russell, Section 20.3, EM algorithm): $\Sigma_{i} = \...
0
votes
0answers
66 views

Likelihood scaling for Bernoulli Mixture Model to avoid underflow

I am very new to Expectation Maximization and struggling with how to scale the likelihood calculations to avoid numeric underflow. I am trying to create a Bernoulli Mixture Model for sparse, binary ...
0
votes
0answers
19 views

Confusion over expectation maximisation algorithm

first of all, apologies if this is the wrong place for this. I've been reading around my actuarial studies, and came across the expectation maximisation algorithm. I first read this article, which ...
2
votes
1answer
77 views

Why is the prior omitted from this Bayes rule?

I'm trying to understand the EM algorithm. I've found a tutorial on it. It goes like this: Two coins (A & B). 5 rounds of flipping 10 times. We forgot, however, which coin was flipped each round. ...
3
votes
0answers
48 views

Termination Condition(s) for Expectation Maximization

What are good criteria for deciding when to terminate the expectation-maximization algorithm. I know that the idea is that you should terminate when the change in the data log likelihood is "small" ...
0
votes
1answer
40 views

Usefulness of EM algorithm

I wonder how EM make things easier when we are finding the MLE with missing data. Let $Z$ be the complete data, $Y = Y(Z)$ the observed data, and $\theta$ the parameter to be estimated. For the MLE, ...
4
votes
1answer
517 views

Is this a typo/error in Bishop's book

I am currently going through the chapter 9 - Mixture Models and EM from Bishop's book - Pattern Recognition and Machine Learning (2006). I could not understand the maximization step with respect to ...
1
vote
1answer
38 views

Distribution function or density in Mixed Distribution EM

In this calculation https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm#E_step a probability, $P(Z_i = j|X_i=x_i;\theta^{(t)})$ is evaluated using bayes theorem, and then each ...
2
votes
0answers
148 views

Using Naive Bayes classifier for unsupervised learning

I was going through this article to learn about how the EM algorithm can be used to use the Naive Bayes algorithm for unsupervised learning. Suppose we have the following data without labels: 1 0 1 1 ...
2
votes
1answer
114 views

What are some applications of unsupervised HMMs?

Supervised HMMs can be applied to many problems like POS tagging and OCR (optical character recognition). I've learned that HMMs can be trained unsupervisedly using EM (Baum-Welch algorithm), what ...
1
vote
1answer
455 views

Estimating truncation point in Gaussian mixture

I have data modeled as a mixture of two Gaussian distributions. The data is "clipped" i.e., there is data only for values greater than a threshold $t$, even though it is feasible for data to exist in ...
0
votes
2answers
58 views

Why different initial parameters of Expectation–Maximization (EM) result in different clusters? [closed]

I'm having a hard time understanding conceptually why we get different clusters when we start the algorithm with different initial parameters. Can anyone explain the mechanisms behind it to me a ...