Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [expectation-maximization]

An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.

0
votes
0answers
9 views

Using EM algorithm to estimate Markov Switching Vector Autoregression but failed to converge [migrated]

Can anybody help me figure out the problem in my code? I write it to estimate the MSVAR model using the EM algorithm proposed by Hamilton J.D. 1990. Basically There're three parameters to estimate. ...
0
votes
1answer
33 views

When shall we use Expectation Maximization (EM) instead of Maximum Likelihood Estimation (MLE)?

I saw in many articles that EM is an algorithm to do MLE, and we usually use it when a direct MLE is not possible. Can someone tell me what is the meaning of "direct MLE is not possible" (and what ...
0
votes
0answers
3 views

Is every variable with unknown value for a particular example is a valid hidden variable for the Expectation Maximization (EM) algorithm?

Can we say that whatever the random variable represent, if its particular value for a given example is unknown then we can use it as a hidden variable for the Expectation Maximization (EM) algorithm? ...
0
votes
1answer
16 views

Can we say the Expectation Maximization (EM) algorithm is supposed to be used for unsupervised or semi-supervised learning?

From what I read and understood, when we have a discrete hidden variable that we already know its particular value (instead of summing/marginalizing over them) associated with data then it is ...
3
votes
2answers
44 views

EM algorithm for mixture of Gaussians - is it ok to use my updated mu's in my new estimate of Sigma, within a single M-step?

Here is a screenshot from an assignment I am currently working on - these are the Expectation-Maximization update rules for the parameters $\omega$ (latent component "responsibilities"), $\mu$, and $\...
1
vote
1answer
71 views

Questions about the likelihood in probabilities?

Many define the likelihood of the data something like $\prod_{x} p(x|\theta)$ others like $p(x|\theta)$. Is the likelihood defined for one sample point/data element (like one document from a ...
0
votes
0answers
23 views

How to use the Expectation Maximization (EM) algorithm for Part of Speech (POS) tagging?

I know that Viterbi algorithm is used for POS tagging, but I want use EM algorithm for that. The corpus is a set of sentences Xs and their POS tags Ys i.e. a sentence X is a sequence of words $(X_1,...
0
votes
1answer
67 views

Find the derivative w.r.t. matrix normal distribution pdf

We have the pdf of matrix normal distribution for the random matrix $X$ (https://en.wikipedia.org/wiki/Matrix_normal_distribution): However here in my case, $X$ is of a parameter, say $\theta$. So my ...
1
vote
0answers
28 views

Can Expectation-Maximization algorithm estimate parameters other than mean and variance (from a model distribution)?

We know that we can use Expectation-Maximization algorithm to estimate parameters from a Gaussian mixture model, say $\mu$, $\sigma$, and $\phi$ (they are parameters of the Gaussian distributions)as ...
0
votes
0answers
13 views

Expectation Maximization for a 2D Normal Model

I'm working through an example in Richard Duda's Pattern Classification on Expecation Maximization Algorithm. Specifically I'm trying to understand the expectation part, and how the parameters get ...
1
vote
1answer
108 views

Understanding the log-likelihood (score) in scikit-learn GMM

I have been training a GMM (Gaussian Mixture, clustering / unsupervised) on two version of the same dataset: one training with all its features and one training after a PCA truncated to its 2 first ...
0
votes
0answers
12 views

Estimating latent mean and variance for a Gaussian

I have a latent Gaussian model with unknown parameters $\mu$ and $\sigma^2$. I can estimate these parameters using MLE and an EM-ish algorithm. However the solution is not stable; I end up in local ...
1
vote
0answers
30 views

Clarification regarding proof of convergence of online EM

Online EM algorithm was proposed by Olivier Cappé in Link to paper. They assume that complete data likelihood $f(x ; \theta)$ belongs to exponential family i.e. $f(x;\theta) = h(x) \exp \left\lbrace ...
0
votes
0answers
40 views

Baum-Welch (EM) algorithm for non-homogeneous Hidden Markov Models

Is there a way of applying the Baum-Welch (or more general, EM) algorithm for non-homogenous Hidden Markov Models, i.e. if the Markov chain depends on covariates?
0
votes
0answers
10 views

Imposing constraints on observation model in a HMM

I have $N$ observations ($x_1, x_2,.. ,x_N$) from a HMM with $K$ latent states. The M step for computing the observation model $\mu_k$ involves maximizing the expression: $$ L = \sum_{n=1}^{N}{ln \...
0
votes
1answer
93 views

Understanding the details of Expectation Maximization(EM) for estimating the parameters?

When using the Expectation Maximization(EM) for estimating the parameters, every time I came across a different problem I see a totally different representation of the likelihood/Expectation function ...
2
votes
1answer
50 views

Simplification of an expectation

While attempting to simplify a combination of expectations, I'm stuck at a particular term whose simplification I'm unable to deduce. The term to be simplified is: $\mathbb{E}[X^{T}F^{T}FX]$ where $...
2
votes
1answer
47 views

Expectation of Sufficient Statistic

Consider $X \sim B(n,p)$ with pmf $P(X=x) = {{n}\choose{x}} p^x (1-p)^{n-x}$. The general exponential form of an exponential family distribution is $p(x|\theta) = f(x) g(\theta) e^{\phi(\theta)^T T(...
0
votes
0answers
101 views

rewriting ELBO to highlight the role of priors

I am reading this paper which rewrites ELBO. I am stuck in verifying the mathematics used for doing the rewriting. Essentially, the paper writes the KL term involved in ELBO as follows (equations 13 ...
1
vote
0answers
34 views

Stopping criteria for gaussian mixture models

As I can read from the source code of scikit-learn, the stopping criteria for the iterative algorithm of Expectation Maximization (in my case applied to fitting Gaussian mixture models) is to put a ...
0
votes
1answer
60 views

EM algorithm for zero truncated poisson

I find it very difficult to understand the E-step of EM algorithm in zero-truncated poisson example. Can someone explain me (mathematically) how exactly do we estimate the number of our "missing" zero ...
0
votes
0answers
26 views

finding the 'best' set

I came across this simple looking but puzzling question recently. There is a set of N tuples given [(a1,b1), ..., (aN, bN)], where a are real numbers and b are positive real numbers. We need to choose ...
2
votes
0answers
25 views

EM algorithm for mixture models, what happens when you don't know how many mixtures you have?

i am trying to learn something here so i tried to estimate the means of two normal distributions that I created. Here is the code: ...
0
votes
0answers
15 views

Is expectation maximization an approximation algorithm?

Is expectation maximization an approximation algorithm? Does it give the exact solution?
0
votes
0answers
14 views

Initializing structural expectation maximization for learning Bayes net structures

I am using bnlearn in R to learn Bayesian network structures. It has a structural.em method for learning with missing data that ...
0
votes
1answer
43 views

How to use log probabilities in PCA mixture EM algorithm

I'm trying to implement PCA mixtures (Tipping & Bishop 2006 Appendix C) on the Tobomovirus. I'll summarize the mathematical background and algorithm here: For a single PCA model, we assume a ...
0
votes
0answers
13 views

How far can local optimum deviate from the ground truth?

I am in the bioinformatic field and see numerous bioinformatic tools applying heuristic (for example EM) methods to find local optimum solution (for example SciClone). I know that local optimum may ...
0
votes
0answers
104 views

EM Algorithm for Bayesian Networks with missing data

Setting: learning parameters of Bayesian Network (BN) with missing data. Algorithm: Expectation-Maximization. Question: suppose I am in the M-step, and that in the complete data there are no ...
0
votes
0answers
23 views

What is the relationship between the EM-algorithm, forward-backward alrgorithm and Viterbi algorithms for Hidden Markov Model?

I know procedure of viterbi, EM-algorithm, and forward-backward independently for Hidden Markov Model. But what the relationship between them?
1
vote
1answer
48 views

log in the M-step of the EM algorithm

In the M-step of the EM algorithm, you have to maximize the expected log-likelihood of X with respect to z which is: $ \int d z P(Z \mid X, \theta^{old}) \ln P(X \mid Z, \theta)$. Why do we maximize ...
0
votes
1answer
26 views

Expectation of a discrete random variable that is case-defined from other discrete random variables

Background The question arises from the following real-life situation: I buy a newspaper at 3 dollars and sell it at 6 dollars. I know the demand for news paper is a binomial random variable with $n=...
1
vote
1answer
93 views

Reducibility between Gaussian Mixture Models and Gaussian Processes

I am studying gaussian processes and I have already discrete amount of knowledge in gaussian mixture models. I am here to undersrtand if with a gaussian process you can fit a gaussian mixture model. ...
0
votes
0answers
22 views

Epsilon from Bivariate Normal Distribution [duplicate]

I came across the following example from a book. I am given a dataset generated from a bivariate normal distribution: Among the data, there are missing values for the last 20 of x2i (but not for x1i)....
0
votes
1answer
13 views

Consistency between EM clusterings with varying starting point

I have a data set (~9 dimensions) in Weka and am running the EM clusterer with a fixed number of clusters. When changing the seed/initial point, the clusterings are very different. Is this expected? ...
0
votes
0answers
21 views

EM algorithm for factor analysis,the formula of diagonal matrix stuck

I am trying to learn the factor analysis of CS229,the relative lecture note is here:CS229 Lecture note9 I have stucked at the diagonal matrix formula which is at the page 9: What's the derivation of ...
0
votes
0answers
117 views

EM Derivation for Dawid-Skene Model

I am trying to derive the EM update equations for the Dawid-Skene model. Following the notation in Bayesian Classifier Combination by Kim and Ghahramani, $i$ is the index of the data point, $t_i$ is ...
0
votes
0answers
47 views

Linear regression of features inside a hidden Markov model?

I have an interesting little problem which I am trying to attack using HMMs. First, as usual, I am trying to do time-series segmentation/classification using a HMM. But the input to my HMM has an ...
1
vote
1answer
53 views

Estimating truth and confusion matrix from noisy observations with Expectation Maximization?

Suppose we have $m$ sources, each of which noisily observe the same set of $n$ independent events from the outcome set $\{A,B,C\}$. Each source has a confusion matrix, for example for source $i$: $$...
1
vote
0answers
101 views

how can I do online Expectation–maximization algorithm?

em algorithm is usually optimized iteratively between the expectation (construct a lower-bound) and the maximize the likelihood (optimize the lower-bound) to guarantee convergence. However, at each ...
0
votes
0answers
126 views

EM algorithm and AIC criteria

I am using EM algorithm to estimate the model parameters. EM-algorithm iterates until the loglikelihood is converged. After that, I need to compute AIC criteria. As known, AIC is a loglikelihood ...
1
vote
2answers
600 views

Comparing K-Means and Expectation Maximization on the dataset generated - When does K-Means perform better?

I was experimenting with K-Means and Gaussian Mixture Models (Expectation-Maximization) on the data set that I generated. Here is how the plot for two distributions looks like: Since this was ...
1
vote
0answers
24 views

Likelihood Construction for Censored Data

I am trying to understand the Expectation-Maximization algorithm, and was trying to read through this paper by Park and Lee. In section 2, "Likelihood Construction for Censored Data", they mention the ...
2
votes
2answers
328 views

Question about the latent variable in EM algorithm

In mixture models, Expectation maximization algorithm (EM) is a commonly used method to estimate the model parameters. Suppose that I have bivariate mixture model with two mixture components, with ...
1
vote
1answer
39 views

What is an Expectation Maximisation Algorithm for Markov chains?

I'm looking for an algorithm for Expectation Maximisation of a Markov chain. I am aware of the Baum-Welch algorithm for Hidden Markov Models, but I can't find an algorithm for Markov Models that are ...
0
votes
1answer
69 views

The ways to normalize the likelihood in EM algorithm

In Wikepedia it states that: In the simplest cases, normalization of ratings means adjusting values measured on different scales to a notionally common scale, often prior to averaging. And ...
0
votes
0answers
49 views

EM algorithm early stopping

Assume there are a set of latent variables $X$, a set of observed variables $Y$ and some parameters $\Theta$. I am using EM algorithm to compute $X$ and $\theta$. In the E step, it computes $p(X|Y,\...
2
votes
0answers
35 views

Is Expectation-Maximization the right type of analysis (& if so, how)?

I have 118 microbes that I have tested at various concentrations of a drug that is supposed to kill them (0.1875 - 15). However, these microbes have mutations (A-H) which confer resistance to the ...
1
vote
0answers
44 views

Optimal Sequence Problem

Lets says I have 100 people that like to buy item x. They ask me to send them a message every time I have x available to sell. Of the 100 people that like to buy item x: 1) Some people will pay more ...
1
vote
0answers
77 views

MLE for high dimensional $\theta$

I'm estimating a parameter $\theta$ in the context of covariance structure model given by $\Sigma(\theta)$. As an estimator, I use ML and computation is done by fmincon function in Matlab(using sqp ...
4
votes
0answers
282 views

Fisher information matrix in logistic regression

I am self-studying the basics of logistic regression. I came across this sentence: In logistic regression expected and observed information matrixes are equal I am aware that the information ...