Questions tagged [expectation-maximization]
An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.
0
votes
0answers
9 views
Using EM algorithm to estimate Markov Switching Vector Autoregression but failed to converge [migrated]
Can anybody help me figure out the problem in my code? I write it to estimate the MSVAR model using the EM algorithm proposed by Hamilton J.D. 1990. Basically There're three parameters to estimate. ...
0
votes
1answer
33 views
When shall we use Expectation Maximization (EM) instead of Maximum Likelihood Estimation (MLE)?
I saw in many articles that EM is an algorithm to do MLE, and we usually use it when a direct MLE is not possible.
Can someone tell me what is the meaning of "direct MLE is not possible" (and what ...
0
votes
0answers
3 views
Is every variable with unknown value for a particular example is a valid hidden variable for the Expectation Maximization (EM) algorithm?
Can we say that whatever the random variable represent, if its particular value for a given example is unknown then we can use it as a hidden variable for the Expectation Maximization (EM) algorithm?
...
0
votes
1answer
16 views
Can we say the Expectation Maximization (EM) algorithm is supposed to be used for unsupervised or semi-supervised learning?
From what I read and understood, when we have a discrete hidden variable that we already know its particular value (instead of summing/marginalizing over them) associated with data then it is ...
3
votes
2answers
44 views
EM algorithm for mixture of Gaussians - is it ok to use my updated mu's in my new estimate of Sigma, within a single M-step?
Here is a screenshot from an assignment I am currently working on - these are the Expectation-Maximization update rules for the parameters $\omega$ (latent component "responsibilities"), $\mu$, and $\...
1
vote
1answer
71 views
Questions about the likelihood in probabilities?
Many define the likelihood of the data something like $\prod_{x} p(x|\theta)$ others like $p(x|\theta)$. Is the likelihood defined for one sample point/data element (like one document from a ...
0
votes
0answers
23 views
How to use the Expectation Maximization (EM) algorithm for Part of Speech (POS) tagging?
I know that Viterbi algorithm is used for POS tagging, but I want use EM algorithm for that. The corpus is a set of sentences Xs and their POS tags Ys i.e. a sentence X is a sequence of words $(X_1,...
0
votes
1answer
67 views
Find the derivative w.r.t. matrix normal distribution pdf
We have the pdf of matrix normal distribution for the random matrix $X$ (https://en.wikipedia.org/wiki/Matrix_normal_distribution):
However here in my case, $X$ is of a parameter, say $\theta$. So my ...
1
vote
0answers
28 views
Can Expectation-Maximization algorithm estimate parameters other than mean and variance (from a model distribution)?
We know that we can use Expectation-Maximization algorithm to estimate parameters from a Gaussian mixture model, say $\mu$, $\sigma$, and $\phi$ (they are parameters of the Gaussian distributions)as ...
0
votes
0answers
13 views
Expectation Maximization for a 2D Normal Model
I'm working through an example in Richard Duda's Pattern Classification on Expecation Maximization Algorithm. Specifically I'm trying to understand the expectation part, and how the parameters get ...
1
vote
1answer
108 views
Understanding the log-likelihood (score) in scikit-learn GMM
I have been training a GMM (Gaussian Mixture, clustering / unsupervised) on two version of the same dataset: one training with all its features and one training after a PCA truncated to its 2 first ...
0
votes
0answers
12 views
Estimating latent mean and variance for a Gaussian
I have a latent Gaussian model with unknown parameters $\mu$ and $\sigma^2$. I can estimate these parameters using MLE and an EM-ish algorithm. However the solution is not stable; I end up in local ...
1
vote
0answers
30 views
Clarification regarding proof of convergence of online EM
Online EM algorithm was proposed by Olivier Cappé in Link to paper.
They assume that complete data likelihood $f(x ; \theta)$ belongs to exponential family i.e.
$f(x;\theta) = h(x) \exp \left\lbrace ...
0
votes
0answers
40 views
Baum-Welch (EM) algorithm for non-homogeneous Hidden Markov Models
Is there a way of applying the Baum-Welch (or more general, EM) algorithm for non-homogenous Hidden Markov Models, i.e. if the Markov chain depends on covariates?
0
votes
0answers
10 views
Imposing constraints on observation model in a HMM
I have $N$ observations ($x_1, x_2,.. ,x_N$) from a HMM with $K$ latent states. The M step for computing the observation model $\mu_k$ involves maximizing the expression:
$$ L = \sum_{n=1}^{N}{ln \...
0
votes
1answer
93 views
Understanding the details of Expectation Maximization(EM) for estimating the parameters?
When using the Expectation Maximization(EM) for estimating the parameters, every time I came across a different problem I see a totally different representation of the likelihood/Expectation function ...
2
votes
1answer
50 views
Simplification of an expectation
While attempting to simplify a combination of expectations, I'm stuck at a particular term whose simplification I'm unable to deduce.
The term to be simplified is: $\mathbb{E}[X^{T}F^{T}FX]$
where $...
2
votes
1answer
47 views
Expectation of Sufficient Statistic
Consider $X \sim B(n,p)$ with pmf $P(X=x) = {{n}\choose{x}} p^x (1-p)^{n-x}$.
The general exponential form of an exponential family distribution is $p(x|\theta) = f(x) g(\theta) e^{\phi(\theta)^T T(...
0
votes
0answers
101 views
rewriting ELBO to highlight the role of priors
I am reading this paper which rewrites ELBO. I am stuck in verifying the mathematics used for doing the rewriting. Essentially, the paper writes the KL term involved in ELBO as follows (equations 13 ...
1
vote
0answers
34 views
Stopping criteria for gaussian mixture models
As I can read from the source code of scikit-learn, the stopping criteria for the iterative algorithm of Expectation Maximization (in my case applied to fitting Gaussian mixture models) is to put a ...
0
votes
1answer
60 views
EM algorithm for zero truncated poisson
I find it very difficult to understand the E-step of EM algorithm in zero-truncated poisson example. Can someone explain me (mathematically) how exactly do we estimate the number of our "missing" zero ...
0
votes
0answers
26 views
finding the 'best' set
I came across this simple looking but puzzling question recently. There is a set of N tuples given [(a1,b1), ..., (aN, bN)], where a are real numbers and b are positive real numbers. We need to choose ...
2
votes
0answers
25 views
EM algorithm for mixture models, what happens when you don't know how many mixtures you have?
i am trying to learn something here so i tried to estimate the means of two normal distributions that I created. Here is the code:
...
0
votes
0answers
15 views
Is expectation maximization an approximation algorithm?
Is expectation maximization an approximation algorithm? Does it give the exact solution?
0
votes
0answers
14 views
Initializing structural expectation maximization for learning Bayes net structures
I am using bnlearn in R to learn Bayesian network structures. It has a structural.em method for learning with missing data that ...
0
votes
1answer
43 views
How to use log probabilities in PCA mixture EM algorithm
I'm trying to implement PCA mixtures (Tipping & Bishop 2006 Appendix C) on the Tobomovirus. I'll summarize the mathematical background and algorithm here:
For a single PCA model, we assume a ...
0
votes
0answers
13 views
How far can local optimum deviate from the ground truth?
I am in the bioinformatic field and see numerous bioinformatic tools applying heuristic (for example EM) methods to find local optimum solution (for example SciClone). I know that local optimum may ...
0
votes
0answers
104 views
EM Algorithm for Bayesian Networks with missing data
Setting: learning parameters of Bayesian Network (BN) with missing data.
Algorithm: Expectation-Maximization.
Question: suppose I am in the M-step, and that in the complete data there are no ...
0
votes
0answers
23 views
What is the relationship between the EM-algorithm, forward-backward alrgorithm and Viterbi algorithms for Hidden Markov Model?
I know procedure of viterbi, EM-algorithm, and forward-backward independently for Hidden Markov Model. But what the relationship between them?
1
vote
1answer
48 views
log in the M-step of the EM algorithm
In the M-step of the EM algorithm, you have to maximize the expected log-likelihood of X with respect to z which is:
$ \int d z P(Z \mid X, \theta^{old}) \ln P(X \mid Z, \theta)$. Why do we maximize ...
0
votes
1answer
26 views
Expectation of a discrete random variable that is case-defined from other discrete random variables
Background
The question arises from the following real-life situation: I buy a newspaper at 3 dollars and sell it at 6 dollars. I know the demand for news paper is a binomial random variable with $n=...
1
vote
1answer
93 views
Reducibility between Gaussian Mixture Models and Gaussian Processes
I am studying gaussian processes and I have already discrete amount of knowledge in gaussian mixture models. I am here to undersrtand if with a gaussian process you can fit a gaussian mixture model.
...
0
votes
0answers
22 views
Epsilon from Bivariate Normal Distribution [duplicate]
I came across the following example from a book. I am given a dataset generated from a bivariate normal distribution:
Among the data, there are missing values for the last 20 of x2i (but not for x1i)....
0
votes
1answer
13 views
Consistency between EM clusterings with varying starting point
I have a data set (~9 dimensions) in Weka and am running the EM clusterer with a fixed number of clusters. When changing the seed/initial point, the clusterings are very different. Is this expected?
...
0
votes
0answers
21 views
EM algorithm for factor analysis,the formula of diagonal matrix stuck
I am trying to learn the factor analysis of CS229,the relative lecture note is here:CS229 Lecture note9
I have stucked at the diagonal matrix formula which is at the page 9:
What's the derivation of ...
0
votes
0answers
117 views
EM Derivation for Dawid-Skene Model
I am trying to derive the EM update equations for the Dawid-Skene model. Following the notation in Bayesian Classifier Combination by Kim and Ghahramani, $i$ is the index of the data point, $t_i$ is ...
0
votes
0answers
47 views
Linear regression of features inside a hidden Markov model?
I have an interesting little problem which I am trying to attack using HMMs.
First, as usual, I am trying to do time-series segmentation/classification using a HMM. But the input to my HMM has an ...
1
vote
1answer
53 views
Estimating truth and confusion matrix from noisy observations with Expectation Maximization?
Suppose we have $m$ sources, each of which noisily observe the same set of $n$ independent events from the outcome set $\{A,B,C\}$. Each source has a confusion matrix, for example for source $i$:
$$...
1
vote
0answers
101 views
how can I do online Expectation–maximization algorithm?
em algorithm is usually optimized iteratively between the expectation (construct a lower-bound) and the maximize the likelihood (optimize the lower-bound) to guarantee convergence. However, at each ...
0
votes
0answers
126 views
EM algorithm and AIC criteria
I am using EM algorithm to estimate the model parameters. EM-algorithm iterates until the loglikelihood is converged. After that, I need to compute AIC criteria. As known, AIC is a loglikelihood ...
1
vote
2answers
600 views
Comparing K-Means and Expectation Maximization on the dataset generated - When does K-Means perform better?
I was experimenting with K-Means and Gaussian Mixture Models (Expectation-Maximization) on the data set that I generated. Here is how the plot for two distributions looks like:
Since this was ...
1
vote
0answers
24 views
Likelihood Construction for Censored Data
I am trying to understand the Expectation-Maximization algorithm, and was trying to read through this paper by Park and Lee. In section 2, "Likelihood Construction for Censored Data", they mention the ...
2
votes
2answers
328 views
Question about the latent variable in EM algorithm
In mixture models, Expectation maximization algorithm (EM) is a commonly used method to estimate the model parameters. Suppose that I have bivariate mixture model with two mixture components, with ...
1
vote
1answer
39 views
What is an Expectation Maximisation Algorithm for Markov chains?
I'm looking for an algorithm for Expectation Maximisation of a Markov chain.
I am aware of the Baum-Welch algorithm for Hidden Markov Models, but I can't find an algorithm for Markov Models that are ...
0
votes
1answer
69 views
The ways to normalize the likelihood in EM algorithm
In Wikepedia it states that:
In the simplest cases, normalization of ratings means adjusting values
measured on different scales to a notionally common scale, often prior
to averaging.
And ...
0
votes
0answers
49 views
EM algorithm early stopping
Assume there are a set of latent variables $X$, a set of observed variables $Y$ and some parameters $\Theta$. I am using EM algorithm to compute $X$ and $\theta$. In the E step, it computes $p(X|Y,\...
2
votes
0answers
35 views
Is Expectation-Maximization the right type of analysis (& if so, how)?
I have 118 microbes that I have tested at various concentrations of a drug that is supposed to kill them (0.1875 - 15). However, these microbes have mutations (A-H) which confer resistance to the ...
1
vote
0answers
44 views
Optimal Sequence Problem
Lets says I have 100 people that like to buy item x. They ask me to send them a message every time I have x available to sell.
Of the 100 people that like to buy item x:
1) Some people will pay more ...
1
vote
0answers
77 views
MLE for high dimensional $\theta$
I'm estimating a parameter $\theta$ in the context of covariance structure model given by $\Sigma(\theta)$. As an estimator, I use ML and computation is done by fmincon function in Matlab(using sqp ...
4
votes
0answers
282 views
Fisher information matrix in logistic regression
I am self-studying the basics of logistic regression. I came across this sentence:
In logistic regression expected and observed information matrixes are
equal
I am aware that the information ...
