Questions tagged [expectation-maximization]
An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.
0
votes
0answers
18 views
latent variables in EM algorithm are assumed to be i.i.d from multinomial distribution, from what they are idependent
In EM algorithm we introduce a latent variables, say $z_i$, $i=1,...n$, $n$ is the number of the mixture component. These variables ($z_i$) are assumed to be independent and identically distributed ...
-1
votes
0answers
12 views
Em algorithm for mixture binomial problems [on hold]
I'm trying to create an em algorithm for mixture binomial problems but my code has a mistake that I can't find.
...
0
votes
0answers
16 views
Baum-Welch algorithm and link to EM algorithm [duplicate]
What is the exact link between Baum-Welch in HMM and EM?
In EM, we usually calculate:
$$Q(X) = P(X|Y,\theta)$$
and then maximize
$$\mathbb{E}_{~Q}[ log(\frac{P(Y, X, \theta)}{Q(X)})].$$
I read the ...
-1
votes
0answers
8 views
Implementation of Expectation - Maximisation Algorithm
I am trying to implement expectation maximisation algorithm for a multivariate Bernoulli model applied to the following file of binary digits:
http://www.gatsby.ucl.ac.uk/teaching/courses/ml1-2017/...
0
votes
0answers
11 views
Is there a way to choose the number of iterations in the EM (expectation maximization) algorithm when doing a Kalman Filter?
I understand that for large datasets it may be slow to implement. Other than that, is there a "usual" or accepted way to choose the number of iterations when estimating parameters in the Kalman Filter,...
0
votes
0answers
11 views
How is jaccard similarity used to find the similarity between Bootstrap samples when measuring stability of EM?
Im reading the answer on "how to determine number of clusters in EM algorithm". How to tell if data is "clustered" enough for clustering algorithms to produce meaningful results?
One of the ...
1
vote
0answers
18 views
Is expectation maximization an example of empirical Bayes?
I don't think I truly understand what methods are classified as "empirical Bayes". Is expectation maximization considered an example of this?
1
vote
1answer
44 views
Can we use a mixture of normal distributions while optimising likelihood?
Let's assume that we generate some values by a mixture of two Gaussians. Now we want to find the parameters of the two Gaussians by likelihood maximisation. One good expect that the optimisation will ...
0
votes
2answers
55 views
Confusion with the E-step of the EM algorithm for Gaussian Mixture Models
So I was reviewing the E-step for the Gaussian Mixture model on Wikipedia.
And it looks like in the E-step all you really need to compute is the conditional distribution of Z because that is all that ...
1
vote
0answers
26 views
EM Algorithm for Poisson Gamma
I would like to check if I have done this question right. I am trying to derive the EM algorithm for $\mu$ in the following distributions:
$$f(Y|Z) = \frac{z^y}{y!} e^{-z} $$
$$f(Z) = \frac{\theta^\...
1
vote
1answer
52 views
When shall we use Expectation Maximization (EM) instead of Maximum Likelihood Estimation (MLE)?
I saw in many articles that EM is an algorithm to do MLE, and we usually use it when a direct MLE is not possible.
Can someone tell me what is the meaning of "direct MLE is not possible" (and what ...
0
votes
0answers
4 views
Is every variable with unknown value for a particular example is a valid hidden variable for the Expectation Maximization (EM) algorithm?
Can we say that whatever the random variable represent, if its particular value for a given example is unknown then we can use it as a hidden variable for the Expectation Maximization (EM) algorithm?
...
1
vote
1answer
20 views
Can we say the Expectation Maximization (EM) algorithm is supposed to be used for unsupervised or semi-supervised learning?
From what I read and understood, when we have a discrete hidden variable that we already know its particular value (instead of summing/marginalizing over them) associated with data then it is ...
3
votes
2answers
53 views
EM algorithm for mixture of Gaussians - is it ok to use my updated mu's in my new estimate of Sigma, within a single M-step?
Here is a screenshot from an assignment I am currently working on - these are the Expectation-Maximization update rules for the parameters $\omega$ (latent component "responsibilities"), $\mu$, and $\...
1
vote
1answer
75 views
Questions about the likelihood in probabilities?
Many define the likelihood of the data something like $\prod_{x} p(x|\theta)$ others like $p(x|\theta)$. Is the likelihood defined for one sample point/data element (like one document from a ...
0
votes
0answers
38 views
How to use the Expectation Maximization (EM) algorithm for Part of Speech (POS) tagging?
I want to know how can we use the EM algorithm for Part of Speech (POS) tagging.
The data is a set of sentences Xs and their POS tags Ys i.e. a sentence X is a sequence of words $(X_1,X_2,\ldots, ...
0
votes
1answer
72 views
Find the derivative w.r.t. matrix normal distribution pdf
We have the pdf of matrix normal distribution for the random matrix $X$ (https://en.wikipedia.org/wiki/Matrix_normal_distribution):
However here in my case, $X$ is of a parameter, say $\theta$. So my ...
1
vote
0answers
47 views
Can Expectation-Maximization algorithm estimate parameters other than mean and variance (from a model distribution)?
We know that we can use Expectation-Maximization algorithm to estimate parameters from a Gaussian mixture model, say $\mu$, $\sigma$, and $\phi$ (they are parameters of the Gaussian distributions)as ...
0
votes
0answers
22 views
Expectation Maximization for a 2D Normal Model
I'm working through an example in Richard Duda's Pattern Classification on Expecation Maximization Algorithm. Specifically I'm trying to understand the expectation part, and how the parameters get ...
1
vote
1answer
352 views
Understanding the log-likelihood (score) in scikit-learn GMM
I have been training a GMM (Gaussian Mixture, clustering / unsupervised) on two version of the same dataset: one training with all its features and one training after a PCA truncated to its 2 first ...
0
votes
0answers
14 views
Estimating latent mean and variance for a Gaussian
I have a latent Gaussian model with unknown parameters $\mu$ and $\sigma^2$. I can estimate these parameters using MLE and an EM-ish algorithm. However the solution is not stable; I end up in local ...
1
vote
0answers
30 views
Clarification regarding proof of convergence of online EM
Online EM algorithm was proposed by Olivier Cappé in Link to paper.
They assume that complete data likelihood $f(x ; \theta)$ belongs to exponential family i.e.
$f(x;\theta) = h(x) \exp \left\lbrace ...
0
votes
0answers
52 views
Baum-Welch (EM) algorithm for non-homogeneous Hidden Markov Models
Is there a way of applying the Baum-Welch (or more general, EM) algorithm for non-homogenous Hidden Markov Models, i.e. if the Markov chain depends on covariates?
0
votes
0answers
12 views
Imposing constraints on observation model in a HMM
I have $N$ observations ($x_1, x_2,.. ,x_N$) from a HMM with $K$ latent states. The M step for computing the observation model $\mu_k$ involves maximizing the expression:
$$ L = \sum_{n=1}^{N}{ln \...
0
votes
1answer
105 views
Understanding the details of Expectation Maximization(EM) for estimating the parameters?
When using the Expectation Maximization(EM) for estimating the parameters, every time I came across a different problem I see a totally different representation of the likelihood/Expectation function ...
2
votes
1answer
52 views
Simplification of an expectation
While attempting to simplify a combination of expectations, I'm stuck at a particular term whose simplification I'm unable to deduce.
The term to be simplified is: $\mathbb{E}[X^{T}F^{T}FX]$
where $...
2
votes
1answer
62 views
Expectation of Sufficient Statistic
Consider $X \sim B(n,p)$ with pmf $P(X=x) = {{n}\choose{x}} p^x (1-p)^{n-x}$.
The general exponential form of an exponential family distribution is $p(x|\theta) = f(x) g(\theta) e^{\phi(\theta)^T T(...
0
votes
0answers
111 views
rewriting ELBO to highlight the role of priors
I am reading this paper which rewrites ELBO. I am stuck in verifying the mathematics used for doing the rewriting. Essentially, the paper writes the KL term involved in ELBO as follows (equations 13 ...
1
vote
0answers
85 views
Stopping criteria for gaussian mixture models
As I can read from the source code of scikit-learn, the stopping criteria for the iterative algorithm of Expectation Maximization (in my case applied to fitting Gaussian mixture models) is to put a ...
0
votes
1answer
95 views
EM algorithm for zero truncated poisson
I find it very difficult to understand the E-step of EM algorithm in zero-truncated poisson example. Can someone explain me (mathematically) how exactly do we estimate the number of our "missing" zero ...
0
votes
0answers
26 views
finding the 'best' set
I came across this simple looking but puzzling question recently. There is a set of N tuples given [(a1,b1), ..., (aN, bN)], where a are real numbers and b are positive real numbers. We need to choose ...
2
votes
0answers
30 views
EM algorithm for mixture models, what happens when you don't know how many mixtures you have?
i am trying to learn something here so i tried to estimate the means of two normal distributions that I created. Here is the code:
...
0
votes
0answers
17 views
Is expectation maximization an approximation algorithm?
Is expectation maximization an approximation algorithm? Does it give the exact solution?
0
votes
0answers
15 views
Initializing structural expectation maximization for learning Bayes net structures
I am using bnlearn in R to learn Bayesian network structures. It has a structural.em method for learning with missing data that ...
0
votes
1answer
51 views
How to use log probabilities in PCA mixture EM algorithm
I'm trying to implement PCA mixtures (Tipping & Bishop 2006 Appendix C) on the Tobomovirus. I'll summarize the mathematical background and algorithm here:
For a single PCA model, we assume a ...
0
votes
0answers
15 views
How far can local optimum deviate from the ground truth?
I am in the bioinformatic field and see numerous bioinformatic tools applying heuristic (for example EM) methods to find local optimum solution (for example SciClone). I know that local optimum may ...
0
votes
0answers
118 views
EM Algorithm for Bayesian Networks with missing data
Setting: learning parameters of Bayesian Network (BN) with missing data.
Algorithm: Expectation-Maximization.
Question: suppose I am in the M-step, and that in the complete data there are no ...
0
votes
0answers
37 views
What is the relationship between the EM-algorithm, forward-backward alrgorithm and Viterbi algorithms for Hidden Markov Model?
I know procedure of viterbi, EM-algorithm, and forward-backward independently for Hidden Markov Model. But what the relationship between them?
1
vote
1answer
53 views
log in the M-step of the EM algorithm
In the M-step of the EM algorithm, you have to maximize the expected log-likelihood of X with respect to z which is:
$ \int d z P(Z \mid X, \theta^{old}) \ln P(X \mid Z, \theta)$. Why do we maximize ...
0
votes
1answer
26 views
Expectation of a discrete random variable that is case-defined from other discrete random variables
Background
The question arises from the following real-life situation: I buy a newspaper at 3 dollars and sell it at 6 dollars. I know the demand for news paper is a binomial random variable with $n=...
1
vote
1answer
94 views
Reducibility between Gaussian Mixture Models and Gaussian Processes
I am studying gaussian processes and I have already discrete amount of knowledge in gaussian mixture models. I am here to undersrtand if with a gaussian process you can fit a gaussian mixture model.
...
0
votes
0answers
22 views
Epsilon from Bivariate Normal Distribution [duplicate]
I came across the following example from a book. I am given a dataset generated from a bivariate normal distribution:
Among the data, there are missing values for the last 20 of x2i (but not for x1i)....
0
votes
1answer
13 views
Consistency between EM clusterings with varying starting point
I have a data set (~9 dimensions) in Weka and am running the EM clusterer with a fixed number of clusters. When changing the seed/initial point, the clusterings are very different. Is this expected?
...
0
votes
0answers
25 views
EM algorithm for factor analysis,the formula of diagonal matrix stuck
I am trying to learn the factor analysis of CS229,the relative lecture note is here:CS229 Lecture note9
I have stucked at the diagonal matrix formula which is at the page 9:
What's the derivation of ...
0
votes
0answers
147 views
EM Derivation for Dawid-Skene Model
I am trying to derive the EM update equations for the Dawid-Skene model. Following the notation in Bayesian Classifier Combination by Kim and Ghahramani, $i$ is the index of the data point, $t_i$ is ...
0
votes
0answers
90 views
Linear regression of features inside a hidden Markov model?
I have an interesting little problem which I am trying to attack using HMMs.
First, as usual, I am trying to do time-series segmentation/classification using a HMM. But the input to my HMM has an ...
1
vote
1answer
59 views
Estimating truth and confusion matrix from noisy observations with Expectation Maximization?
Suppose we have $m$ sources, each of which noisily observe the same set of $n$ independent events from the outcome set $\{A,B,C\}$. Each source has a confusion matrix, for example for source $i$:
$$...
1
vote
0answers
148 views
how can I do online Expectation–maximization algorithm?
em algorithm is usually optimized iteratively between the expectation (construct a lower-bound) and the maximize the likelihood (optimize the lower-bound) to guarantee convergence. However, at each ...
0
votes
0answers
167 views
EM algorithm and AIC criteria
I am using EM algorithm to estimate the model parameters. EM-algorithm iterates until the loglikelihood is converged. After that, I need to compute AIC criteria. As known, AIC is a loglikelihood ...
1
vote
2answers
848 views
Comparing K-Means and Expectation Maximization on the dataset generated - When does K-Means perform better?
I was experimenting with K-Means and Gaussian Mixture Models (Expectation-Maximization) on the data set that I generated. Here is how the plot for two distributions looks like:
Since this was ...
