Questions tagged [expectation-maximization]
An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.
562
questions
0
votes
0
answers
14
views
Hidden Markov Model - Baum Welch algorithm initialization
Currently I'm working on a problem where I have a multidimensional, continuous sequence of observations $X$ that model my response variable $y$ with two states $0$ and $1$. I assume that this sequence ...
- 249
0
votes
0
answers
14
views
How to conduct EM algorithm when there are some outliers in GMM Models?
I'm just confused about the problem of adding an outlier component directly to the primary form of GMM models:
Suppose that the observed data contains several outliers. The mixture model could be:
$$
...
- 1
0
votes
1
answer
19
views
Conditional independence in EM algorithm
Let $X$, $\theta$ and $Z$ denote observed, parameter and latent nodes in a graphical model. The EM algorithm attempts to find a local maximum likelihood estimate $\theta^\ast$ for the likelihood of ...
- 212
3
votes
1
answer
91
views
Is the likelihood for Gaussian mixture models still multimodal when Y is partially observed?
In discussing Gaussian mixture models (GMMs), https://normaldeviate.wordpress.com/2012/08/04/mixture-models-the-twilight-zone-of-statistics/ highlights the issue of
Multimodality of the Likelihood. ...
- 3,768
1
vote
1
answer
39
views
Issue with Casella&Berger derivation of EM likelihood equality
In the explanation of the EM (Expectation maximization) algorithm p.328 in the book "Statistical inference" by G. Casella and R. Berger, 2nd edition, they present the following:
$\mathbf{Y} =...
- 357
0
votes
0
answers
12
views
Expectation-Maximization high missing rates and multiple variables
I know MICE can be used for imputation of multiple variables simultaneously. The expectation maximization approach (EM) can be used to impute missing data. Typically, one should only be using ...
- 73
0
votes
0
answers
18
views
Labeling unlabeled data (expectation maximization)
Say I have a database (Excel) consisting of 10k different dresses and accompanying attributes (column names) for each dress (sleeve length, color, pattern, ...). I would like to label each of these ...
0
votes
1
answer
78
views
How does expectation maximization compute missing data?
I have been searching for a simple example of how expectation-maximization (EM) computes missing data. All the examples I have found are based on multivariate normal models. I have seen that EM can be ...
1
vote
0
answers
46
views
Problems with convergence of the EM algorithm for a gaussian mixture regression
I have been implementing a EM-algorithm for a latent-class regression model, where every individual has a vector of observations.
Currenly, I have the problem that the model does not converge. The log ...
- 21
0
votes
0
answers
55
views
EM derivation for mixture of t distributions
(Notation is at the bottom.)
I'm currently self-studying the Expectation-Maximization algorithm and has learned to derive it for various mixture models including Gaussian mixture models, mixture of ...
- 124
0
votes
0
answers
9
views
Model choice with expectation maximization: which likelihood?
When deciding about the number of mixture components using Akaike or bayesian information criteria, should one use the full likelihood or the likelihood marginalized over the latent variables? Both ...
- 1,578
0
votes
0
answers
37
views
Benefits of Expectation Maximization for Mixture Models
What are the benefits of using expectation maximization for mixture models vs. direct maximization of the marginal likelihoods?
Analytic maximization step
In case of Gaussian mixtures the benefit is ...
- 1,578
0
votes
1
answer
91
views
Computing a prior from two components in Naive Bayes
Given a model parameter $\theta$ that is composed of two distributions in a Naive Bayes classifier, how is $P(\theta)$ typically computed in practice?
More specifically, from the article of Nigam et ...
- 212
0
votes
0
answers
40
views
Expectation maximization: does the likelihood always increase monotonically?
When working with (gaussian) mixture models, I always took it for a mathematical fact that the marginal likelihood increases with every iteration step. If it were not the case, it always meant an ...
- 1,578
0
votes
0
answers
46
views
EM algorithm for multinomial data
Good afternoon ,
Assume we have a dataset modeled by multinomial data of K cluster.
To apply EM algorithm parameters are :
$${\theta }^{\left(0\right)}=({p}_{k}^{\left(0\right)},k=1,\mathrm{\dots },K)$...
- 113
2
votes
0
answers
42
views
Can the EM algorithm be used to perform model selection?
Suppose that the samples $\mathcal{D} = \{x_1,x_2,\dots,x_N\}$ were independently sampled from a Gaussian distribution with mean $\mu$ and variance $\sigma^2$. However, suppose that we did not know ...
- 2,916
0
votes
0
answers
10
views
Residual Diagnostics of an Approximate Dynamic Factor Model estimated through EM-Algorithm
I am estimating and forecasting macroeconomic variables using an approximate dynamic factor model using EM-Algorithm. The model allows the idiosyncratic terms to follow an AR(1) process.
So in OLS ...
- 109
1
vote
1
answer
156
views
Derivation of the M-step in EM algorithm for a three-dimensional panel mixture model
I have a question regarding the estimation of a latent-class gaussian mixture model, where the model is for three dimensional panel data set with individuals $i$, in country $j$ in time $t$.
I want ...
- 21
0
votes
0
answers
28
views
Applying EM Algorithm to 3x3 Contingency Table
Given the contingency table
$$
\begin{array}{l}
\begin{array}{l|ccc}
& \text { Alcholoic } & \text {Non-Alcholic} & \text { Not Answered } \\
\hline \text { Male } & n_{11} & n_{...
- 1
2
votes
2
answers
35
views
Practical considerations for EM clustering
EM algorithm guarantees finding a local rather then global minimum of the likelihood. As a consequence, the results are dependent on the initial conditions (e.g., if randomly choosing the initial ...
- 1,578
1
vote
1
answer
354
views
How can I derive the EM algorithm for a mixture of two Bernoulli distributions?
How can I derive the E-step and M-step in the EM algorithm for a mixture of two Bernoulli distributions? Note that I am aware that there are several notes online that explain how to do this for the ...
- 2,916
15
votes
2
answers
666
views
Finding category with maximum likelihood method
Let's say that we had an information for men and women heights.
R code:
...
- 153
4
votes
1
answer
92
views
Why use the EM Algorithm and not just marginalise the complete likelihood?
On the wikipedia article for Expectation-Maximization it states
Given the statistical model which generates a set $\mathbf{X}$ of observed data, a set of unobserved latent data or missing values $\...
- 61
0
votes
0
answers
14
views
EM (?) algorithm with dependent observations but the prior seems not informative
How can we derive the (EM?) algorithm to estimate $b$ if we suppose
$G_j = bg_j$ for all $j$,
the priors (?) of the observations $\hat{G}_j \sim N(\hat{G}_j | G_j, (s_{G, j})^2)$, $\hat{g}_j \sim N(\...
- 101
0
votes
1
answer
33
views
EM algorithm when there are too many components to calculate the function Q
Assuming a regression model as follow:
$$\mathbf{y} = \mathbf{x}\beta + \mathbf{\varepsilon}$$
where
$\mathbf{y}=(y_1,...,y_n)^T\in\mathbb{R}^{n\times 1}$,
$\mathbf{x}=(x_1,...,x_n)^T\in\mathbb{R}^{n\...
- 1
1
vote
2
answers
108
views
Is this interpretation of the EM algorithm correct?
Suppose we want to estimate the parameter $\theta$ of the distribution $p(z;\theta)$ of a random variable $Z$. If we had the i.i.d samples $z_1, z_2, \dots, z_N$, we could easily do this using maximum ...
- 2,916
0
votes
0
answers
28
views
Maximum likelihood expectation maximization (for image segmentation)
I am trying to wrap my head around the concept of using Maximum likelihood expectation maximization for image segmentation. I understand the concept of maximum likelihood as a way of finding the most ...
- 13
1
vote
1
answer
48
views
EM algorithm and random effects
I am trying to learn how to fit random effects/mixed effects models. I read on this article that quasi-likelihood, numerical integration and MCMC are the most common methods.
My question is:
can we ...
- 2,536
0
votes
0
answers
26
views
Reference Request: Variational Expectation-Maximization algorithm for Latent Dirichlet Allocation with an added time component
This link has a pretty good runthrough on the variational inference (via variational E-M) for LDA with calculations expanded and explained.
I am now considering a modified LDA which adds a time ...
- 1
4
votes
1
answer
429
views
K Means as a special case of GMM (using EM Algorithm)
I am looking for a tutorial/gentle introduction (preferably with mathematics/proofs) on K-means as a special case of Gaussian Mixture Model using the EM Algorithm.
I have found this: https://www....
- 7,545
1
vote
1
answer
69
views
Understanding numerical example of expectation maximization
I was trying to understand Expectation maximization algorithm. This is how it is defined in Andrew Ng's Stanford CS229 course:
$$
\text{Repeat until convergence \{}\quad\quad\quad\quad\quad\quad\...
- 123
0
votes
0
answers
21
views
Expected 0-1 under multinomial distribution
Suppose the world generates a single observation $x \sim \text{multinomial}(\theta)$, where the parameter vector $\theta = (\theta_1,\ldots, \theta_k)$ with $\theta_i\geq 0$ and $\theta_1+\ldots+\...
- 121
2
votes
1
answer
172
views
Is there a special case of the EM algorithm for exponential family distributions?
According to Wikipedia, the formal definition of the EM algorithm is
The EM algorithm seeks to find the MLE of the marginal likelihood by iteratively applying these two steps:
Expectation step (E ...
- 2,916
0
votes
0
answers
51
views
Areas of research in statistical machine learning
I'm reading from a book called Machine Learning: A Probabilistic Perspective by Kevin Murphy. Besides being somewhat challenging to understand, I feel that the earlier chapters (on probability, ...
0
votes
1
answer
32
views
Penalized model based clustering variable selection with L1 penalty
Ok so I am working through a paper by Pan and Shen where they maximize a penalized version of the complete data log-likelihood
$$
Q_{P}\left(\Theta ; \Theta^{(m)}\right)=E_{\Theta^{(m)}}\left(\log L_{...
- 93
0
votes
0
answers
12
views
In theory, does the form of the latent dynamics matter?
I am experimenting with latent variable models of the form
$\mathbf{x}_t = \mathbf{W}\mathbf{z}_t + \mathcal{E}_t$ where $\mathcal{E}_t \sim \mathcal{N}(0,\sigma^2\mathbf{I})$ and $\mathbf{z}_t \sim \...
- 101
0
votes
1
answer
49
views
Learning HMM parameters by counting?
In 8.4.3 of the book Speech and Language Processing: An introduction to natural language processing, the two matrices transition probabilities and emission probabilities can be learned by counting as ...
- 5,151
0
votes
1
answer
60
views
Jensen's inequality in EM derivation: expectation derivation
In Where does Jensen's Inequality come into the EM derivation? and in McLachlan & Krishnan (1997) - The EM Algorithm and Extensions, it was shown that
\begin{equation}
H(\theta|\theta^{(t)}) \...
- 155
0
votes
0
answers
12
views
Are the subsets of Mixture Models necessarily parametrized?
Im just learning about mixture models and they are described as a "mix of parametric and non parametric models".
My question is how a non-parametric subset would look like? How would e.g. EM-...
- 1
0
votes
1
answer
119
views
Monte Carlo approximation to find expected value of gradient square
I need to to calculate this term:
$
\mathbb{E}\left[S(Y, L,\theta)S(Y,L,\theta)^\prime\right]
$
Where
$
S(Y,L,\theta) =\frac{\partial}{\partial\theta}
l(Y,L,\theta)
$
With $\theta$ = maximum ...
- 25
0
votes
0
answers
26
views
Multivariate Normal Underflow
Good day everyone
At the moment I am attempting to write code in R to calculate the following.
$$
\tau_{k j}^{(m)}=\frac{\pi_{k}^{(m)} f_{k}\left(x_{j} ; \theta_{k}^{(m)}\right)}{f\left(x_{j} ; \Theta^...
- 93
1
vote
1
answer
80
views
Expected log likelihood for mixture components with differing support
I was hoping to use the EM algorithm to fit a mixture model in which the mixture components can have differing support. I've run into a problem during the M step because the expected log-likelihood ...
- 13
1
vote
0
answers
27
views
High dimensional multivariate normal calculations in R
Good day everyone
At the moment I am attempting to write code in R to calculate the following.
$$
\tau_{k j}^{(m)}=\frac{\pi_{k}^{(m)} f_{k}\left(x_{j} ; \theta_{k}^{(m)}\right)}{f\left(x_{j} ; \Theta^...
- 93
2
votes
1
answer
77
views
linear regression inappropriate when there are multiple underlying groups
In this scenario, one group has a clear linear relationship between x and y. Another group doesn't. However, there is no way to differentiate them in the data. In this case, performing a simple linear ...
- 193
0
votes
0
answers
173
views
EM algorithm for multivariate gaussian with diagonal covariance matrix
Ok so quick question.
Say I need to use the EM-algorithm to estimate the parameters of a multivariate gaussian
$$
f_{k}\left(x ; \theta_{k}\right)=\frac{1}{(2 \pi)^{P / 2}|V|} \exp \left(-\frac{1}{2}\...
- 93
0
votes
1
answer
74
views
What are the variations of Expectation Maximization?
To explain my question better, I will use this analogy:
In the case of the Gradient-Descent method, we have multiple variations/expansions for the main algorithm, like stochastic gradient descent (SGD)...
- 65
2
votes
2
answers
99
views
Why is $q(\mathbf{z})$ chosen to be the posterior distribution in the EM algorithm?
In the CS229 Lecture Notes on the EM algorithm by Tengyu Ma and Andrew Ng (2019), the authors write that
$$
\log(p(\mathbf{x};\theta)) = \log\left(\mathbb{E}_{q(\mathbf{z})}\left[\frac{p(\mathbf{x},\...
- 2,916
0
votes
0
answers
74
views
EM Algorithm for Kalman Filters
Say I have the following dynamical system with unknown covariances for $w,v$.
$$
z_n = Az_{n-1}+w\\
x_n = Bz_n +v
$$
where $w \sim \mathcal{N}(0,Q)$ and $v \sim \mathcal{N}(0,R)$.
I want to apply the ...
- 177
0
votes
0
answers
25
views
Why isn't this term considered an expectation in EM algorithm?
The log-likelihood $\mathcal{L}$ for a mixture model can be written as:
$$ \mathcal{L} = \log p(\boldsymbol X| \Delta, \boldsymbol \pi) = \sum_{n=1}^N \log \color{blue}{\sum_{k=1}^K \pi_k p(x_n|\...
- 113
0
votes
1
answer
41
views
Regression where the matrix is drawn from a known distribution
I need to estimate the vector x, where Ax = b, given b is observed data and the columns in the matrix A are drawn from a known distribution. a and b are both formed of positive integers (or zero).
For ...
- 1