Questions tagged [expectation-maximization]

An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.

Filter by
Sorted by
Tagged with
0 votes
0 answers
15 views

Expectation Maximization on Multivariate Gaussian Mixture Model for clustering

I have a dataset with 1000 observations and two features that define those N=1000 data points. Hence it is 1000*2 input matrix. I need to cluster them into k clusters. I am not understanding the E-M ...
Oindri's user avatar
  • 1
0 votes
0 answers
30 views

Q function in Baum-Welch algorithm

Assume there's a hidden Markov Model $\lambda=(\pi, \mathbf{A}, \mathbf{B})$, where $\pi$ is an initial distribution, $\mathbf{A}$ is a transition matrix and $\mathbf{B}$ is an emission matrix. Also, ...
thesecond's user avatar
  • 298
2 votes
1 answer
91 views

How can one estimate the parameters of unobserved variables?

Consider three discrete variables $X, Y , Z$ where $$X \sim B(p_x)$$ $$P(Y = 1 |X = 1) = p_y,\ P(Y = 0 |X = 1) = 1 - p_y,\ P(Y = 1 |X = 0) = 0$$ I think I can say that $$Y \sim B(p_xp_y)$$ And $$S = ...
Knz's user avatar
  • 23
0 votes
1 answer
13 views

Determine the parameters of a particle filter that best fit observations

I am wondering is there any established framework to optimize the parameter $\lambda$ of a particle filter such that $p(O|\lambda)$ is maximized, where $O$ is the observation sequence. For HMM and ...
Geng Wang's user avatar
1 vote
1 answer
37 views

The Tsallis entropy of generalized Gaussian distribution

I would like to discuss the computation of the Tsallis entropy for the generalized Gaussian distribution. From the paper in the link https://www.sciencedirect.com/science/article/pii/S0167947322000822....
M.cadirci's user avatar
0 votes
0 answers
30 views

EM algorithm get new parameters by optimizing the Q function (lower bound of likelihood function) or optimizing the likelihood function

We know that in the EM (Expectation-Maximization) algorithm, the E-step determines the $Q$ function by calculating expectations, which is a lower bound of the likelihood function. In the M-step, by ...
yi ren's user avatar
  • 1
0 votes
0 answers
31 views

How to compute the covariance matrix for a mixture model estimated by the EM algorithm

I am trying to compute the observed Fisher information matrix for a mixture model estimated by the EM algorithm. My original thought is to simply compute the second derivative of a mixture density. ...
Lydia2kkx's user avatar
0 votes
0 answers
10 views

number of parameters in Dirichlet Mixture Model clustering (non-bayesian)

I made a function that implements the clustering algorithm in the research article "Clustering compositional data using Dirichlet mixture model" (2022). I am now trying to figure out which ...
Immanuel Kunt's user avatar
1 vote
0 answers
16 views

Which Fisher information to use to obtain Cramer-Rao bound in expectation-maximization?

I have a rather limited understanding of statistical estimation theory so I apologize if my question is strange or trivial. Say I have an expectation-maximization-based algorithm for determining the ...
DangerousTim's user avatar
0 votes
0 answers
37 views

How to deal with the high dimensionality when using EM algorithm to solve Gaussian mixture models?

When I use the EM algorithm to solve a Gaussian mixture model, we may encounter the computation of Gaussian densities in the E step. Specifically, we should have the posterior probability as $$ \pi_{...
Lei's user avatar
  • 1
0 votes
0 answers
61 views

EM algorithm "maximization step" for product of two independent Bernoulli variables

We are given the following model, where the event $C=1$ is defined using two other Bernoulli variables $E$ and $R$: $$P(C=1|q,d,k)=P(E=1|k)\cdot P(R=1|q,d)$$ Which we denote as: $$\theta_k=P(E=1|k), \...
Noral Gata's user avatar
0 votes
1 answer
22 views

Fitting a Gaussian Mixture Model with known share of noise/outliers

A Gaussian Mixture model is fitted by the Expectation-Maximization algorithm. This fairly simple iterative algorithm consists of two steps and the initialization. Initialization (for ...
Klops's user avatar
  • 156
0 votes
0 answers
23 views

Baum Welch algorithm for Semi-Continuous (Tied-Mixture) Hidden Markov Model

I'm currently implementing Baum-Welch algorithm for Semi-Continuous Hidden Markov Model (SCHMM) as described in Huang, Xuedong. Semi-continuous hidden Markov models for speech recognition, 1989, pages ...
thesecond's user avatar
  • 298
1 vote
1 answer
65 views

Why complete data log likelihood in M-step of EM algorithm

In Bishop's Pattern Recognition and Machine Learning book(page440), it talks about the M-step in EM algorithm of Gaussian Mixture Model. I am confused about the likelihood function of M-step. By ...
user3153824's user avatar
0 votes
2 answers
39 views

How to understand the binary latent variable z in GMM model?

GMM(Gaussian Mixture Model) itself is a mixture of Gaussian with each having the proportion of $\pi_k$, $$\sum_{k=1}^{K}\pi_k=1$$this is easy to understand. But when introducing the latent, I don't ...
user3153824's user avatar
0 votes
0 answers
19 views

Understanding Monte Carlo EM in a mixed effects model context

I am currently reading Mixed Effects Models for Complex Data by Lang Wu. On page 136, the author mentions the Monte Carlo EM algorithm as a way of treating missing covariate in the mixed effect model. ...
Faithhhhhh's user avatar
0 votes
0 answers
61 views

Issue: SPSS computes impossible data with Expectation Maximization for missing values

I am a masterstudent in behavioural sciences, currently writing my thesis. Unfortunately statistics isn't my biggest talent, so I was hoping someone out here could help me with an issue I am facing. I ...
Elsbeth Brouwers's user avatar
0 votes
1 answer
67 views

Why is the E-step in the EM algorithm called this way?

Yes, this has been asked before here, but for different reasons. In the E-Step nothing is calculated, we simply define the function, yet once it is defined it is defined once and for all. We could ...
timtam's user avatar
  • 203
1 vote
0 answers
67 views

Is the EM algorithm guaranteed to converge if the log likelihood is concave

As the EM algorithm is guaranteed to increase the log likelihood at each iteration. If the log likelihood is concave is it guaranteed to converge to the maximum of the likelihood, that is will we get ...
Dylan Dijk's user avatar
0 votes
1 answer
63 views

How should I analyze longitudinal data with missing observations for individuals? Expectation Maximization or panel data analysis or regression? How?

I possess a dataset containing 1,000 chickens. For each chicken, we have egg count data for various sizes (11 to 22mm) and the overall number of chicks that hatched from these eggs for each chiken. ...
insan's user avatar
  • 85
0 votes
0 answers
27 views

How to draw dimension reduced high dimensional gaussians in 2D for EM algorithm visualization

I'm implementing the EM algorithm. The visualization works for 2D features. I'd like to visualize it for higher dimensional data using dimension reduction(PCA) Here k= 3. Each group of elipses are ...
Houou In Kyouma's user avatar
2 votes
0 answers
14 views

MNAR Imputation On Time Series Data in Ongoing Study

I am working on an ongoing study that tracks subjects through time. Each subject is enrolled for months, where upon exiting I calculate their length-of-stay. Since the study is ongoing, I have both ...
MVH's user avatar
  • 21
0 votes
0 answers
41 views

Dirichlet Mixture Model manually

I am implementing the Dirichlet Mixture Model using the EM algorithm in R, but am experiencing issues with the results. I generated two binomial distributions with fractions of (70%, 30%) and means of ...
hollyjolly's user avatar
0 votes
0 answers
26 views

Help me express a tractable complete data log likelihood of this problem with hidden variables

Imagine the following scenario: a couple of friends(total nr of people=7) will all play 5 games. At each game the player will ask an horiscope if he should use one out of two bernoulli distributions. ...
anton's user avatar
  • 1
0 votes
0 answers
12 views

How to efficiently caclulate probability of a state in HMM when a random shuffle operation happens on emitted observations

The problem setup is as follows(it's from a book and may not be tied to reality): Suppose we have some speakers s_1, s_2..s_k seated around a table speaking at ...
Bug Killer's user avatar
0 votes
0 answers
64 views

can EM algorithm be used to improve on binary classification?

Just to give some background, I am currently using data(this is mock data) with a binary target as shown below. My binary classification model is based of features 1 to 4 with target variable. ID1 ...
Pontificator's user avatar
2 votes
1 answer
48 views

MLE for a mixture of betas without using EM algorithm

Suppose I have a mixture of two Beta densities say $f_1 = \text{Beta}(1,1)$ and $f_2= \text{Beta}(1,\beta)$ where $\beta$ is unknown. The sample $X_1,....,X_n$ is observed based on latent Bernoulli ...
Maths Freak's user avatar
0 votes
0 answers
29 views

Can we get a Global Maxima by using EM algorithm? [duplicate]

Actually, I read many articles and blogs that EM converges to local maxima can someone help me that it converges to local maxima always? Can we get a global maximum?
RAJ SHIVAM's user avatar
7 votes
1 answer
231 views

EM algorithm for Bivariate Normal

Consider a random sample $X_i = (U_i,V_i)$ where $i=1,2,...,n$ from a bivariate normal population with mean $(\mu_1,\mu_2)$ and variances $(\sigma_1 ^2, \sigma_2 ^2)$ and correlation $\rho$. Let's ...
Maths Freak's user avatar
0 votes
0 answers
20 views

Choosing action that maximize expectation but with uncertain estimates

Hi CrossValidated community, I am looking to understand how to adjust the typical bayesian decision theory to a situation when probability estimates have high uncertainty. In my specific case I am <...
dev85's user avatar
  • 13
0 votes
0 answers
28 views

EM algorithm for finding discrete latent variables with gaussian child in Python

I have the following Bayesian network model: $P(A, B, C, D) = P(D|B)P(C|B)P(B|A)P(A)$ A and B are discrete variables, while C and D are continuous variables. I've looked around and it seems in a ...
Ian's user avatar
  • 61
0 votes
0 answers
28 views

Compute Mutual Information from a fixed dataset

The goal is to compute the mutual information between two variables from the fixed dataset. $$I(X;Z)= E_{P_{XZ}}[T(X,Z)]-log(E_{P_X\otimes P_Z}[e^{T(X,Z)}])$$ This equation is taken from this paper. ...
Justlife's user avatar
  • 103
0 votes
1 answer
81 views

Could the likelihood increase monotonically in a misspecified EM algorithm?

I am dealing with the estimation of a Gaussian Hidden Markov Model with conditional distribution given the first-order Markov state $S_t = j,\ j=1,...,J$ $$ Y_t|S_t=j\sim N(0,\sigma^2_j) $$ where the ...
Mr Frog's user avatar
  • 329
1 vote
0 answers
402 views

EM algorithm on a mixture of two uniforms?

I am having serious issues understanding the EM algorithm, both the E and the M steps when it comes to a mixture of two uniform distributions. I am given the pdf of the mixture which is: $f(x)= \frac{...
user369704's user avatar
0 votes
0 answers
66 views

Hierarchical Expectation Maximization

Let us assume we have a generative model $y = g(\theta,x)$, where $\theta$ is a set of parameters, and $x$ is a set of latent variables. Given a datapoint $y$, I need to update the parameters $\theta$ ...
Alfred's user avatar
  • 325
1 vote
0 answers
16 views

Kinds of Expectation-Maximization [closed]

I'm starting to dig into the variational inference literature, as I need to solve a learning problem where the latent variables form an hierarchical structure (an hierarchical Gaussian generative ...
Alfred's user avatar
  • 325
3 votes
1 answer
161 views

Maximum likelihood for mixture of Bernoullis with known mixture proportions

Given the hierarchical model $$ \begin{align} k & \sim \text{Categorical}(\pi_1, \dots, \pi_K) \\ X \mid k & \sim \text{Bernoulli}(\theta_{k}) \end{align} $$ and an i.i.d. sample $X_1, \dots, ...
tddevlin's user avatar
  • 3,337
1 vote
0 answers
24 views

How to get (Gaussian) mean and covariance of missing variables in the E-Step of the EM Algorithm for Missing Data [duplicate]

The M step for missing data imputation using MVN updates $$\mu^t=\frac{1}{N_D}\sum_n E[y_n]\\ \Sigma^t=\frac{1}{N_d}\sum_nE[y_n y_n^\top]-\mu^t(\mu^t)^\top $$ where $y_n=(x_n,z_n )$. However, the ...
wd violet's user avatar
  • 601
1 vote
0 answers
27 views

EM equality constraint using Jensen's Inequality

I'm trying to derive the ELBO as per CS229 Page 144, Eq 11.8. For E-step, the LHS and RHS must be equal: $$ \begin{align} f\left(\mathbb{E}_{z\sim Q}\left[ \frac{p(x, z; \theta)}{Q(z)}\right]\right) \...
Karan Shah's user avatar
0 votes
0 answers
18 views

Observation from each cluster from EM algorithm manually implemented

I am interesting in implementing the EM algorithm manually. And I did it. However, I am still do not know how to get the observation from each cluster? For example, after run the EM algorithm, the ...
Alice's user avatar
  • 640
4 votes
1 answer
80 views

Need help understanding why the original proof of the EM algorithm had a flaw in it

I understand that in the original paper on the EM algorithm by Dempster, the proof of convergence in Theorem 2 had a flaw, and that a full proof wasn't given until CF Jeff Wu published it years later. ...
user321627's user avatar
  • 3,482
0 votes
0 answers
63 views

Continuously running the M step while approximating the E step

Let us assume we need to maximize the likelihood $p(x|\theta)$ of an observed variable $x$ given a set of parameters $\theta$. However, this likelihood also depends on a set of hidden variables $z$, ...
Alfred's user avatar
  • 325
1 vote
1 answer
81 views

details of stochastic Expectation Maximization (EM) algorithm

I went into a paper, Online EM Algorithm for Latent Data Models (Olivier Cappé & Eric Moulines, 2009). I got confused by the first equation the authors wrote, the Q function: Here the authors ...
C.Wang's user avatar
  • 11
0 votes
0 answers
68 views

Can overlap impact the EM algorithm?

I am using EM algorithm for clustering. I see that if there is an overlap, the accuracy is reduced. However, it is improvesd if no overlap exists. My question is, can overlap affect the performance of ...
Alice's user avatar
  • 640
5 votes
0 answers
123 views

How does the EM algorithm work in Bayesian regression? [closed]

I have problem distinguishing between the latent variables $z_i$ and the parameters $\theta_i$ in EM algorithm. Suppose we have the hierarchical priors \begin{aligned} \beta|\tau,\omega &\sim \...
Isaac 's user avatar
  • 199
2 votes
0 answers
175 views

E-Step of the EM algorithm on mixture

Im trying to implement the EM algorithm on mixture model: $pg(x) + (1-p)h(x)$ where the sample $\bar{x} = (x_1, \ldots, x_n)$, is independently generated from the mixture, and $g(x) = e^{-x}$ and $h(x)...
Occhima's user avatar
  • 161
2 votes
1 answer
137 views

EM algorithm on discrete random variable

Im trying to find the iteration formula for the maximum likelihood estimator of the parameter $\theta$ of the following discrete random variable, through the EM algorithm: $$P(Y = y ) = \sum_{i=0}^{\...
Occhima's user avatar
  • 161
0 votes
1 answer
39 views

What is 'factored models'?

What is 'factored models'? I've seen the term 'factored models' in paper 'The Bayesian Structural EM Algorithm'. But I see it for the first time and it is hard to understand... I tried searching it ...
vinsh_77's user avatar
0 votes
0 answers
67 views

Solve the EM algorithm of the Gaussian mixture model

Considering a mixture of two normally distributed random variables $Y_1$ and $Y_2$ and considering the linear combination of the two: $$ \begin{gathered} Y_{1} \sim N\left(\mu_{1}, \sigma_{1}^{2}\...
JoZ's user avatar
  • 649
0 votes
0 answers
55 views

How to conduct EM algorithm when there are some outliers in GMM Models?

I'm just confused about the problem of adding an outlier component directly to the primary form of GMM models: Suppose that the observed data contains several outliers. The mixture model could be: $$ ...
Iris88's user avatar
  • 1

1
2 3 4 5
12