Questions tagged [expectation-maximization]
An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.
589
questions
0
votes
0
answers
15
views
Expectation Maximization on Multivariate Gaussian Mixture Model for clustering
I have a dataset with 1000 observations and two features that define those N=1000 data points. Hence it is 1000*2 input matrix. I need to cluster them into k clusters.
I am not understanding the E-M ...
- 1
0
votes
0
answers
30
views
Q function in Baum-Welch algorithm
Assume there's a hidden Markov Model $\lambda=(\pi, \mathbf{A}, \mathbf{B})$, where $\pi$ is an initial distribution, $\mathbf{A}$ is a transition matrix and $\mathbf{B}$ is an emission matrix. Also, ...
- 298
2
votes
1
answer
91
views
How can one estimate the parameters of unobserved variables?
Consider three discrete variables $X, Y , Z$ where
$$X \sim B(p_x)$$
$$P(Y = 1 |X = 1) = p_y,\ P(Y = 0 |X = 1) = 1 - p_y,\ P(Y = 1 |X = 0) = 0$$ I think I can say that $$Y \sim B(p_xp_y)$$
And
$$S = ...
- 23
0
votes
1
answer
13
views
Determine the parameters of a particle filter that best fit observations
I am wondering is there any established framework to optimize the parameter $\lambda$ of a particle filter such that $p(O|\lambda)$ is maximized, where $O$ is the observation sequence. For HMM and ...
- 11
1
vote
1
answer
37
views
The Tsallis entropy of generalized Gaussian distribution
I would like to discuss the computation of the Tsallis entropy for the generalized Gaussian distribution. From the paper in the link https://www.sciencedirect.com/science/article/pii/S0167947322000822....
- 85
0
votes
0
answers
30
views
EM algorithm get new parameters by optimizing the Q function (lower bound of likelihood function) or optimizing the likelihood function
We know that in the EM (Expectation-Maximization) algorithm, the E-step determines the $Q$ function by calculating expectations, which is a lower bound of the likelihood function. In the M-step, by ...
- 1
0
votes
0
answers
31
views
How to compute the covariance matrix for a mixture model estimated by the EM algorithm
I am trying to compute the observed Fisher information matrix for a mixture model estimated by the EM algorithm. My original thought is to simply compute the second derivative of a mixture density. ...
- 11
0
votes
0
answers
10
views
number of parameters in Dirichlet Mixture Model clustering (non-bayesian)
I made a function that implements the clustering algorithm in the research article "Clustering compositional data using Dirichlet mixture model" (2022). I am now trying to figure out which ...
1
vote
0
answers
16
views
Which Fisher information to use to obtain Cramer-Rao bound in expectation-maximization?
I have a rather limited understanding of statistical estimation theory so I apologize if my question is strange or trivial.
Say I have an expectation-maximization-based algorithm for determining the ...
- 11
0
votes
0
answers
37
views
How to deal with the high dimensionality when using EM algorithm to solve Gaussian mixture models?
When I use the EM algorithm to solve a Gaussian mixture model, we may encounter the computation of Gaussian densities in the E step. Specifically, we should have the posterior probability as
$$
\pi_{...
- 1
0
votes
0
answers
61
views
EM algorithm "maximization step" for product of two independent Bernoulli variables
We are given the following model, where the event $C=1$ is defined using two other Bernoulli variables $E$ and $R$:
$$P(C=1|q,d,k)=P(E=1|k)\cdot P(R=1|q,d)$$
Which we denote as:
$$\theta_k=P(E=1|k), \...
0
votes
1
answer
22
views
Fitting a Gaussian Mixture Model with known share of noise/outliers
A Gaussian Mixture model is fitted by the Expectation-Maximization algorithm.
This fairly simple iterative algorithm consists of two steps and the initialization.
Initialization (for ...
- 156
0
votes
0
answers
23
views
Baum Welch algorithm for Semi-Continuous (Tied-Mixture) Hidden Markov Model
I'm currently implementing Baum-Welch algorithm for Semi-Continuous Hidden Markov Model (SCHMM) as described in Huang, Xuedong. Semi-continuous hidden Markov models for speech recognition, 1989, pages ...
- 298
1
vote
1
answer
65
views
Why complete data log likelihood in M-step of EM algorithm
In Bishop's Pattern Recognition and Machine Learning book(page440), it talks about the M-step in EM algorithm of Gaussian Mixture Model. I am confused about the likelihood function of M-step. By ...
- 49
0
votes
2
answers
39
views
How to understand the binary latent variable z in GMM model?
GMM(Gaussian Mixture Model) itself is a mixture of Gaussian with each having the proportion of $\pi_k$, $$\sum_{k=1}^{K}\pi_k=1$$this is easy to understand. But when introducing the latent, I don't ...
- 49
0
votes
0
answers
19
views
Understanding Monte Carlo EM in a mixed effects model context
I am currently reading Mixed Effects Models for Complex Data by Lang Wu. On page 136, the author mentions the Monte Carlo EM algorithm as a way of treating missing covariate in the mixed effect model.
...
- 323
0
votes
0
answers
61
views
Issue: SPSS computes impossible data with Expectation Maximization for missing values
I am a masterstudent in behavioural sciences, currently writing my thesis. Unfortunately statistics isn't my biggest talent, so I was hoping someone out here could help me with an issue I am facing.
I ...
0
votes
1
answer
67
views
Why is the E-step in the EM algorithm called this way?
Yes, this has been asked before here, but for different reasons. In the E-Step nothing is calculated, we simply define the function, yet once it is defined it is defined once and for all. We could ...
- 203
1
vote
0
answers
67
views
Is the EM algorithm guaranteed to converge if the log likelihood is concave
As the EM algorithm is guaranteed to increase the log likelihood at each iteration. If the log likelihood is concave is it guaranteed to converge to the maximum of the likelihood, that is will we get ...
- 319
0
votes
1
answer
63
views
How should I analyze longitudinal data with missing observations for individuals? Expectation Maximization or panel data analysis or regression? How?
I possess a dataset containing 1,000 chickens. For each chicken, we have egg count data for various sizes (11 to 22mm) and the overall number of chicks that hatched from these eggs for each chiken.
...
- 85
0
votes
0
answers
27
views
How to draw dimension reduced high dimensional gaussians in 2D for EM algorithm visualization
I'm implementing the EM algorithm. The visualization works for 2D features. I'd like to visualize it for higher dimensional data using dimension reduction(PCA)
Here k= 3. Each group of elipses are ...
2
votes
0
answers
14
views
MNAR Imputation On Time Series Data in Ongoing Study
I am working on an ongoing study that tracks subjects through time. Each subject is enrolled for months, where upon exiting I calculate their length-of-stay.
Since the study is ongoing, I have both ...
- 21
0
votes
0
answers
41
views
Dirichlet Mixture Model manually
I am implementing the Dirichlet Mixture Model using the EM algorithm in R, but am experiencing issues with the results. I generated two binomial distributions with fractions of (70%, 30%) and means of ...
0
votes
0
answers
26
views
Help me express a tractable complete data log likelihood of this problem with hidden variables
Imagine the following scenario: a couple of friends(total nr of people=7) will all play 5 games.
At each game the player will ask an horiscope if he should use one out of two bernoulli distributions. ...
- 1
0
votes
0
answers
12
views
How to efficiently caclulate probability of a state in HMM when a random shuffle operation happens on emitted observations
The problem setup is as follows(it's from a book and may not be tied to reality):
Suppose we have some speakers s_1, s_2..s_k seated around a table speaking at ...
- 101
0
votes
0
answers
64
views
can EM algorithm be used to improve on binary classification?
Just to give some background, I am currently using data(this is mock data) with a binary target as shown below.
My binary classification model is based of features 1 to 4 with target variable. ID1 ...
2
votes
1
answer
48
views
MLE for a mixture of betas without using EM algorithm
Suppose I have a mixture of two Beta densities say $f_1 = \text{Beta}(1,1)$ and $f_2= \text{Beta}(1,\beta)$ where $\beta$ is unknown. The sample $X_1,....,X_n$ is observed based on latent Bernoulli ...
- 217
0
votes
0
answers
29
views
Can we get a Global Maxima by using EM algorithm? [duplicate]
Actually, I read many articles and blogs that EM converges to local maxima can someone help me that it converges to local maxima always? Can we get a global maximum?
7
votes
1
answer
231
views
EM algorithm for Bivariate Normal
Consider a random sample $X_i = (U_i,V_i)$ where $i=1,2,...,n$ from a bivariate normal population with mean $(\mu_1,\mu_2)$ and variances $(\sigma_1 ^2, \sigma_2 ^2)$ and correlation $\rho$. Let's ...
- 217
0
votes
0
answers
20
views
Choosing action that maximize expectation but with uncertain estimates
Hi CrossValidated community,
I am looking to understand how to adjust the typical bayesian decision theory to a situation when probability estimates have high uncertainty. In my specific case I am <...
- 13
0
votes
0
answers
28
views
EM algorithm for finding discrete latent variables with gaussian child in Python
I have the following Bayesian network model:
$P(A, B, C, D) = P(D|B)P(C|B)P(B|A)P(A)$
A and B are discrete variables, while C and D are continuous variables. I've looked around and it seems in a ...
- 61
0
votes
0
answers
28
views
Compute Mutual Information from a fixed dataset
The goal is to compute the mutual information between two variables from the fixed dataset.
$$I(X;Z)= E_{P_{XZ}}[T(X,Z)]-log(E_{P_X\otimes P_Z}[e^{T(X,Z)}])$$
This equation is taken from this paper.
...
- 103
0
votes
1
answer
81
views
Could the likelihood increase monotonically in a misspecified EM algorithm?
I am dealing with the estimation of a Gaussian Hidden Markov Model with conditional distribution given the first-order Markov state $S_t = j,\ j=1,...,J$
$$
Y_t|S_t=j\sim N(0,\sigma^2_j)
$$
where the ...
- 329
1
vote
0
answers
402
views
EM algorithm on a mixture of two uniforms?
I am having serious issues understanding the EM algorithm, both the E and the M steps when it comes to a mixture of two uniform distributions.
I am given the pdf of the mixture which is:
$f(x)= \frac{...
- 11
0
votes
0
answers
66
views
Hierarchical Expectation Maximization
Let us assume we have a generative model $y = g(\theta,x)$, where $\theta$ is a set of parameters, and $x$ is a set of latent variables. Given a datapoint $y$, I need to update the parameters $\theta$ ...
- 325
1
vote
0
answers
16
views
Kinds of Expectation-Maximization [closed]
I'm starting to dig into the variational inference literature, as I need to solve a learning problem where the latent variables form an hierarchical structure (an hierarchical Gaussian generative ...
- 325
3
votes
1
answer
161
views
Maximum likelihood for mixture of Bernoullis with known mixture proportions
Given the hierarchical model
$$
\begin{align}
k & \sim \text{Categorical}(\pi_1, \dots, \pi_K) \\
X \mid k & \sim \text{Bernoulli}(\theta_{k})
\end{align}
$$
and an i.i.d. sample $X_1, \dots, ...
- 3,337
1
vote
0
answers
24
views
How to get (Gaussian) mean and covariance of missing variables in the E-Step of the EM Algorithm for Missing Data [duplicate]
The M step for missing data imputation using MVN updates
$$\mu^t=\frac{1}{N_D}\sum_n E[y_n]\\
\Sigma^t=\frac{1}{N_d}\sum_nE[y_n y_n^\top]-\mu^t(\mu^t)^\top
$$
where $y_n=(x_n,z_n )$.
However, the ...
- 601
1
vote
0
answers
27
views
EM equality constraint using Jensen's Inequality
I'm trying to derive the ELBO as per CS229 Page 144, Eq 11.8.
For E-step, the LHS and RHS must be equal:
$$
\begin{align}
f\left(\mathbb{E}_{z\sim Q}\left[ \frac{p(x, z; \theta)}{Q(z)}\right]\right) \...
- 113
0
votes
0
answers
18
views
Observation from each cluster from EM algorithm manually implemented
I am interesting in implementing the EM algorithm manually. And I did it. However, I am still do not know how to get the observation from each cluster? For example, after run the EM algorithm, the ...
- 640
4
votes
1
answer
80
views
Need help understanding why the original proof of the EM algorithm had a flaw in it
I understand that in the original paper on the EM algorithm by Dempster, the proof of convergence in Theorem 2 had a flaw, and that a full proof wasn't given until CF Jeff Wu published it years later.
...
- 3,482
0
votes
0
answers
63
views
Continuously running the M step while approximating the E step
Let us assume we need to maximize the likelihood $p(x|\theta)$ of an observed variable $x$ given a set of parameters $\theta$. However, this likelihood also depends on a set of hidden variables $z$, ...
- 325
1
vote
1
answer
81
views
details of stochastic Expectation Maximization (EM) algorithm
I went into a paper, Online EM Algorithm for Latent Data Models (Olivier Cappé & Eric Moulines, 2009). I got confused by the first equation the authors wrote, the Q function:
Here the authors ...
- 11
0
votes
0
answers
68
views
Can overlap impact the EM algorithm?
I am using EM algorithm for clustering. I see that if there is an overlap, the accuracy is reduced. However, it is improvesd if no overlap exists.
My question is, can overlap affect the performance of ...
- 640
5
votes
0
answers
123
views
How does the EM algorithm work in Bayesian regression? [closed]
I have problem distinguishing between the latent variables $z_i$ and the parameters $\theta_i$ in EM algorithm. Suppose we have the hierarchical priors
\begin{aligned}
\beta|\tau,\omega &\sim \...
- 199
2
votes
0
answers
175
views
E-Step of the EM algorithm on mixture
Im trying to implement the EM algorithm on mixture model: $pg(x) + (1-p)h(x)$ where the sample $\bar{x} = (x_1, \ldots, x_n)$, is independently generated from the mixture, and $g(x) = e^{-x}$ and $h(x)...
- 161
2
votes
1
answer
137
views
EM algorithm on discrete random variable
Im trying to find the iteration formula for the maximum likelihood estimator of the parameter $\theta$ of the following discrete random variable, through the EM algorithm:
$$P(Y = y ) = \sum_{i=0}^{\...
- 161
0
votes
1
answer
39
views
What is 'factored models'?
What is 'factored models'?
I've seen the term 'factored models' in paper 'The Bayesian Structural EM Algorithm'.
But I see it for the first time and it is hard to understand...
I tried searching it ...
- 63
0
votes
0
answers
67
views
Solve the EM algorithm of the Gaussian mixture model
Considering a mixture of two normally distributed random variables $Y_1$ and $Y_2$ and considering the linear combination of the two:
$$
\begin{gathered}
Y_{1} \sim N\left(\mu_{1}, \sigma_{1}^{2}\...
- 649
0
votes
0
answers
55
views
How to conduct EM algorithm when there are some outliers in GMM Models?
I'm just confused about the problem of adding an outlier component directly to the primary form of GMM models:
Suppose that the observed data contains several outliers. The mixture model could be:
$$
...
- 1