Questions tagged [expectation-maximization]
An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.
471
questions
1
vote
0answers
15 views
Example of Parameter-Expanded EM (PX-EM) algorithm
Can someone provide me some paper or book with and example of the PX-EM algorithm?
I'm kind of understanding the theory, but having big trouble understanding how to put it in practice...
In "The EM ...
1
vote
1answer
11 views
Fitting mixture model on data with duplicate values
What is the correct procedure to fit finite mixture models on data with many duplicate values using EM? Let's say I have N(0,1) distributed data and try to fit a 2 component mixture using EM. There ...
1
vote
0answers
20 views
Confusion in Sampling using the IP algorithm (Bishop PRML)
I'm reading Bishop's PRML p. 537 and I don't understand one piece of the IP (data augmentation) algorithm. Namely, the part that says "we use the samples $\{\mathbf{Z}^{(l)}\}$ obtained in the I step ...
0
votes
0answers
12 views
Newton-Raphson method to solve for dof when performing MLE of a multivariate Student-t distribution using EM
I am reading the derivation of EM algorithm to estimate the maximum likelihood of a multivariate Student-t distribution $\mathcal{T}(\mathbf{x} \vert \pmb{\mu}, \pmb{\Sigma}, \nu)$ in Kevin Murphy's ...
0
votes
0answers
40 views
Why Expectation and Maximization algorithm not used in Machine Learning while Gradient Descent algorithm used in Machine Learning?
I know that Newton Raphson, Expectation & Maximization, and Gradient Descent are all known to be optimization methods. Somehow, I wonder why Gradient Descent is chosen to be used in most of ...
0
votes
0answers
15 views
Do I impute missing values with the response?
I have a dataset with missing values in both predictors and the response. As far as I know, the data are missing not at random, so I cannot simply use listwise deletion. Instead, I employed the EM ...
2
votes
0answers
16 views
Can someone verify if the following Bayesian Information Criterion (BIC) model selection algorithm is correct for Gaussian mixture models?
I am trying to find an automated way of picking the number of clusters $K \in \mathbb{N}$ for unsupervised learning scenarios, specifically for GMM.
I was suggested to use something called the "...
4
votes
0answers
35 views
Expectation Maximisation (EM) Algorithm
Some of my parameters do not have a closed form solution. Thus, for these parameters the M-step is implemented via a one-step Newton-Raphson update, i.e.,
\begin{equation}
\theta^{t+1} = \theta^t - \...
0
votes
0answers
11 views
Outlier detection with EM
I am interested in using expectation maximization for outlier detection. In the literature this is usually done assuming that the data of interest are normally distributed while the outliers are ...
0
votes
0answers
11 views
Using expectation maximization for robust regression
What are the advantages/disadvantages of using EM for robust estimation vs. the robust estimation with Huber or Tukey loss functions?
0
votes
0answers
26 views
assessing the stability of importance (sampling) weights
I have read that when importance weights are used, the stability (variability) of the weights should be assessed (Levine and Casella, 2001) -- however, I wonder how this might be accomplished.
For ...
2
votes
2answers
47 views
EM-algorithm for two clusters (when one of the distributions is uniform)
I am having a hard time with the EM-algorithm. Here's the problem that I am trying to solve.
Dealing with noisy annotations is a common problem in computer vision, especially when using ...
3
votes
2answers
73 views
Accounting for uncertain information (few observations) in a prior (empirial Bayes)
I did not really know how to choose an adequate title for this question, so please feel free to change it.
I have a weird case wherein frequentist and Bayesian philosophies come together. I am ...
1
vote
0answers
19 views
Expectation Maximisation vs Expectation Propagation in the context of Bayesian Networks
I am confused about Expectation Maximisation and Expectation Propagation algorithms in the context of Bayesian Networks, especially whether one comprise another.
What is the difference between ...
1
vote
0answers
8 views
Consistency of EM for missing data in non-parametric setting
When we have missing data, a parametric model, and an expectation-maximization procedure, and we want to show that our procedure leads to consistent estimators, we can sometimes set up score functions ...
1
vote
1answer
36 views
Calculation of AIC in finite mixture modeling
I have a question about calculation the AIC to find my optimal amount of clusters. I am applying mixture modeling with the EM algorithm. I know the formula AIC = -2ln(log-lik) + 2k.
These are my log-...
0
votes
0answers
10 views
Analysing faithful dataset in R using GMM
I have got a project on analysing the faithful data in R found in the package "datasets" and called using data(faithful) which is the data set off eruption time and waiting time of the Old Faithful ...
1
vote
0answers
18 views
Are my data a good candidate for EM imputation followed by exploratory factor analysis?
I am doing Exploratory Factor Analysis (EFA) in R, using principal axis factoring in the psych package. I have missing data that prevent me getting factor scores, so I am imputing data. I am using ...
0
votes
0answers
18 views
Cosine Similarity for Classification to EM Cluster?
Perhaps my question sounds naive, uncovering the very little knowledge that I have in the field of Statistics, but is very urgent to get a solid answer or trigger for further insights for my concerns.
...
1
vote
0answers
39 views
A trivial question about EM algorithm theory
In "The EM Algorithm and Extensions", second edition, from Geoffrey J. McLachlan and Thriyambakam Krishnan, X is the latent variable, and Y is de observed (incomplete) variable
I'm little confuse ...
1
vote
1answer
56 views
Using AIC/BIC within cross-validation for likelihood based loss functions
For a course I am teaching, I am having my students fit a Gaussian mixture model using MLEs via the EM algorithm to a bivariate dataset. I have asked the students to use use cross-validation to choose ...
0
votes
0answers
21 views
Compound distribution
I'm trying to compute a maximum likelihood of compound Poisson exp distribution in R by using EM-algorithm method. The distribution is defined by ∑Nj=1Yj where Yn is i.i.d sequence independent exp(θ) ...
1
vote
0answers
25 views
EM algorithm, Elements of Statistical Learning, expectation of log-likelihood
While I am reading ESL, I have some questions in chapter 8 (Model inference and averaging). Specifically,8.5.2 The EM algorithm in general. This part explains how EM works, in general, referring to ...
0
votes
0answers
58 views
Expectation Maximization (EM) stopping criterion
In several EM description (e.g., Theory and Use of the EM Algorithm By Maya R. Gupta and Yihua Chen) I read that two tipical stopping criterions for EM are defined on the difference between log-...
2
votes
1answer
13 views
Visualized Imputation of 2D Sample Space With Saturation Corresponding to Confidence?
I want to visualize a 2D sample space imputed from a 2D scatter of measurements. Something like a 2D hexbin color histogram would work if it were enhanced in the ...
1
vote
0answers
21 views
Is there a way to run constrained EM?
I would like to run an EM (expectation maximization) algorithm to estimate hidden Markov model parameters except, in my case, I have an extra constraint on my start and final states in my HMM.
How do ...
1
vote
1answer
44 views
State Space models: rewriting the Likelihood to estimate the covariance matrix
I have a State Space model
$
\begin{matrix}
Y_t & = & FX_t +R_{t}^{1/2}\epsilon_t \\
X_{t+1} & = & GX_{t}+Du_t
\end{matrix} $
where $\epsilon_t$ and $u_{t}$ distributed ...
0
votes
0answers
8 views
Convergence of Modified Expectation-Maximisation Algorithm - interpreting language of question
We're going to consider a modified E-M algorithm and its convergence properties. To do so, we will first need to review the convergence of the standard E-M algorithm as I'll need to refer back to it. ...
0
votes
0answers
33 views
Convergence of a Expectation Maximisation Algorithm
Consider using standard expectation maximisation to learn the parameters of a Hidden Markov Model. We can show the effect of standard expectation-maximisation on the log-likelihood by first writing ...
0
votes
0answers
18 views
What is an appropriate threshold for the EM algorithm?
I am implementing the Baum-Welch algorithm (special case of the EM algorithm) on a hidden Markov model and I now have to pick an appropriate stopping criteria $\epsilon$ so that the algorithm ...
2
votes
1answer
87 views
Figueiredo and Jain's Gaussian mixture EM convergence criterion
I have implemented and been playing around Figueiredo & Jain 's trainer in this paper http://www.lx.it.pt/~mtf/IEEE_TPAMI_2002.pdf for Gaussian mixture. Fig. 2 in the paper depicts the full ...
0
votes
2answers
46 views
Convergence of EM algorithm
I am aware that EM eventually converges. However, I still have some confusions regarding this property:
1: As far as I am aware, HMM, Gaussian mixture model and MCMC can converge and all of them use ...
1
vote
2answers
94 views
Conceptual explanation of Maximum Likelihood Estimation
Given a generic time-series $$y_{t+1}= \alpha y_{t} + \Sigma_{t+1}^{1/2}\varepsilon_{t+1} \quad \text{with} \quad \varepsilon_{t+1}=N(0, I)$$
where $\Sigma_{t+1}^{1/2}$ indicates the conditional ...
3
votes
1answer
46 views
MLE when variance of residuals is null (y is a linear combination of x)
Suppose I have the following model to be estimated via MLE assuming normal errors $y_{t}=x_{t} \beta +e_{t}$ with $e=N(0,\sigma^{2})$, where $y, x$ are matrixes and $\beta$ is a vector, so $\sigma^{2}$...
0
votes
0answers
22 views
MLE vs Expectation Maximization to estimate time-changing parameters in state space model
Suppose I have a generic model in state-space form described as
$$x_{t+1}=\phi_{t} x_{t}+w_{t+1}\epsilon_{t+1}$$
$$y_{t}=H_{t}x_{t}+v_{t}e_{t+1}$$
where both $e_{t+1}$ and $\epsilon_{t+1}$ are iid ...
0
votes
0answers
25 views
Mixture of Multinomials
I have implemented a Naive Bayes Model for a mixture of independent Bernoulli. Where the conditional probability can be written as:
$\mathbb{P}(Y=j | X) \propto \omega_{j} \prod_{i=1}^{d} \mu_{i, j}^{...
1
vote
0answers
31 views
Robust Expectation-Maximization?
The Expectation-Maximization (EM) algorithm is useful for applying the Maximum Likelihood Estimation (MLE) when there exist latent (hidden) variables in the model. However, when dealing with outliers, ...
1
vote
1answer
53 views
Where does Jensen's Inequality come into the EM derivation?
I am working my way through the original EM paper Maximum Likelihood from Incomplete Data by Dempster, et al.
I have run into a problem with a statement made in section 3. "General Properties". ...
0
votes
0answers
43 views
HMM training of multiple observations corresponding to different hidden states
Given a set of states $\{q_1,q_2,..,q_n\}$, I am considering the following problem. Corresponding to a sequence of hidden states, I have some observations.
first sequence of hidden states ; first ...
0
votes
1answer
22 views
Is there a way to determine the number of the mixture components prior to run EM algorithm
I am working with mixture models. The common way to determine the number of
the mixture components is fitting several mixture models with a different number of mixture components and then select the ...
4
votes
0answers
57 views
Implementation of EM algorithm confusion
Here EM algorithm manually implemented, there's a question of the implementation in R of the EM algorithm for 2 mixed gaussians. The answer has a supposedly correct implementation.
However, don't the ...
1
vote
1answer
248 views
Example of manual implementation of baum-welch algorithm in R
Is there any code out there that implements the baum-welch algorithm for a very basic problem? It would be very helpful to actually see the algorithm in action to better understand how it works.
I ...
0
votes
0answers
31 views
EM algorithm for lognormal with time right censored data
Is there any literature out there that discusses this problem mentioned? I know that it is apparent in survival analysis but I could not find any resources relating to this topic
EDIT: using the EM ...
1
vote
1answer
108 views
MCAR, MAR and EM
I have a binary(1/0) classification task. I am trying to find $p(y = 1 | X)$ where $X$ is the vector of input variables and $y$ is the binary output label.
Suppose that for some records the output ...
0
votes
0answers
39 views
Would estimating a mixture model using MLE or EM provide different results?
I have built a very hard mixture R (code) using EM-algorithm. My supervisor asked me to repeat the estimation using MLE as it is simpler than EM-algorithm. Building another complex R code will take a ...
1
vote
1answer
66 views
EM algorithm and Mean residual life
I am reading Robert Hogg's (Introduction to Mathematical Statistics) EM algorithm.
In example 6.6.1 (page 370 in the 7th version), please help to explain how the following integral
$$\int_a^\infty(...
0
votes
0answers
47 views
latent variables in EM algorithm are assumed to be i.i.d from multinomial distribution, from what they are idependent
In EM algorithm we introduce a latent variables, say $z_i$, $i=1,...n$, $n$ is the number of the mixture component. These variables ($z_i$) are assumed to be independent and identically distributed ...
0
votes
0answers
16 views
Baum-Welch algorithm and link to EM algorithm [duplicate]
What is the exact link between Baum-Welch in HMM and EM?
In EM, we usually calculate:
$$Q(X) = P(X|Y,\theta)$$
and then maximize
$$\mathbb{E}_{~Q}[ log(\frac{P(Y, X, \theta)}{Q(X)})].$$
I read the ...
0
votes
0answers
33 views
How is jaccard similarity used to find the similarity between Bootstrap samples when measuring stability of EM?
Im reading the answer on "how to determine number of clusters in EM algorithm". How to tell if data is "clustered" enough for clustering algorithms to produce meaningful results?
One of the ...
1
vote
0answers
25 views
Is expectation maximization an example of empirical Bayes?
I don't think I truly understand what methods are classified as "empirical Bayes". Is expectation maximization considered an example of this?
