Questions tagged [expectation-maximization]
An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.
521
questions
1
vote
1answer
78 views
EM Algorithm Negative Log Likelihood
I am fitting a shared parameter model where I have a binary $Y_2$ and a continuous $Y_1$, and I am using random effects in the mean structure to model them jointly
$$f(Y_1,Y_2)=\int f(Y_1,Y_2,b)\,\...
0
votes
0answers
12 views
Expectation-Maximization (EM) algorithm with known means
I am trying to fit a mixture of 1D Gaussian distributions to some data. Can the EM algorithm be used in the case of known mean (all the mean values are equal to zero) and fit only the variances of the ...
4
votes
0answers
36 views
Why is the EM algorithm well suited for exponential families?
I've been brushing up on the EM algorithm, and while I feel like I understand the basics, I keep seeing the claim made (e.g. here, here, among several others) that EM works particularly well for ...
0
votes
0answers
28 views
Estimation of joint modeling of binary and continuous random variables via shared parameter
I am trying to fit a joint model of a binary and continuous outcome with repeated measures. I am trying to fit it using a shared parameter model that induces all the correlation between outcome and ...
1
vote
0answers
66 views
What calculation is implied by this Gaussian term?
I'd appreciate this community's help in understanding the calculation implied by the Gaussian term in this equation:
$$w_{ik} = \alpha_{k} \mathcal{N}( \mathbf{y}_i | \mathbf{X}_i \mathbf{\beta}_k , \...
0
votes
0answers
15 views
Imputation methods for time series data (non-stationary)
I am looking for an impute method for non-stationary time series (financial indeces).
From https://pypi.org/project/impyute/ (https://buildmedia.readthedocs.org/media/pdf/impyute/latest/impyute.pdf) I ...
0
votes
0answers
15 views
EM Algorithm For Distributions That Share Parameters
I've implemented an EM algorithm for a gaussian mixture where the mean is defined by an Ornstein–Uhlenbeck process. The pdf I'm trying to maximize is:
$$f(r(s)|r(t)) = (1-q)\cdot e^{-(r(s) - r(t) - k(\...
0
votes
1answer
61 views
Find a lower bound for the minimax risk $\displaystyle \min_{\delta}\max_{\theta\in\Omega}EL(\theta,\delta(X))\ge1-\frac{1}{2^n}$
Consider a random binary vector $X\in\{0,1\}^n. $Let $\theta\in\Omega$ be a probability vector in $\mathbb R^{2^n}$ with $X\sim\theta$. Consider the loss function $\displaystyle L(\theta,a)=\max_{x\in\...
2
votes
0answers
46 views
Optimizing this log-likelihood
I have a HMM which emits an observation Z.
The parameters of the HMM are $\boldsymbol\theta$. $$\boldsymbol\theta = {\boldsymbol{A},\boldsymbol{B},\pi}$$
Where $\boldsymbol{A}$ is the transition ...
3
votes
2answers
187 views
EM algorithm for MLE from a bivariate normal sample with missing data: Stuck on M-step
I'm trying to understand applying the EM algorithm to compute the MLE in a missing data problem.
Specifically, suppose $(x_1,y_1),\ldots,(x_n,y_n)$ is a random sample from the bivariate normal ...
0
votes
0answers
15 views
Multiple imputation using proc mi EM method in SAS
I'd like to clarify how to use SAS for (multiple) imputation in SAS, specifically the EM method option for proc mi. Do I need to analyze multiple imputed sets using proc mianalyze as mentioned in ...
3
votes
1answer
104 views
Bayesian Regression- Expectation Maximization
In Bayesian regression, we have $y_i=x_i^{T}w+\epsilon_i$ where $w \sim \mathcal{N}(0,\alpha)$ and $\epsilon_i \sim \mathcal{N}(0,\frac{1}{\beta})$.
Inference of $\alpha$ and $\beta$ is done by ...
0
votes
0answers
13 views
Statistical Interpretation of 2 data set measures
I was adviced to post here (initialy post on physics exchange but I am going to remove it).
I have two independant experiments have measured $\tau_{1},\sigma_{1}$ and $\tau_{2},\sigma_{2}$ with $\...
0
votes
0answers
36 views
Likelihood of a latent graphical model
What is the approach to take when trying to find the likelihood of the observations on a latent graphical model, with intertwining conditional distributions?
The model:
Each vertex $X_i$ of a binary ...
0
votes
0answers
18 views
Derive expectation of the log determinant of a precision matrix from a Wishart distribution
I'm reading through section 21.6 of Murphy's Machine Learning: A probabilistic perspective where they derive the variational bayes algorithm for fitting a mixture of gaussians. One of the steps ...
0
votes
0answers
17 views
GMM for nonlinear mean
In conventional GMM, observations $\mathbf{X} = \left\lbrace \mathbf{x}_1,\mathbf{x}_2,\ldots\right\rbrace$ are draw from a distribution
$$
\mathbf{x}_n \sim \sum_{k=1}^{K}\pi_k\mathcal{N}\left( \...
1
vote
0answers
32 views
Expectation Maximization Correctness of Problem Formulation
Suppose I draw $n$ iid samples from a Poisson$(\lambda)$ distribution, with $\lambda$ unknown. Now, I artificially turn every 3 I draw into a 1, so that the probability of observing any particular non-...
5
votes
2answers
195 views
Why do we say EM is a partially non-Bayesian method?
I have some difficulties understanding the phrase 'EM is a partially non-Bayesian method'. EM works in iterative fashion. Is it because the iterative nature of EM is somehow similar to prior - ...
0
votes
0answers
30 views
What is the nature of $r_i^t$?
Using Expectation Maximization (EM) algorithm, I want to vary the number of clusters used according to $K = [2,4, ... 50]$ for a normal distribution initialized randomly (...
0
votes
1answer
41 views
Conjectures regarding EM approximations of mixtures of multivariate normal distributions
Consider $X\in\mathbb{R}^{N\times d}$ containing data for $N$ points in $d$ dimensions drawn from a bimodal multivariate normal distribution, where any row $x$ of $X$ follows the mixed multivariate ...
2
votes
0answers
39 views
How does maximising ELBO for a Gaussian mixture model fit the model to data?
I am following along in Bishop's Pattern Recognition and ML chapters 9 and 10, and I understand that the EM algorithm works by iteratively updating model parameters using equations derived from ...
0
votes
0answers
15 views
Making sense of the belief propagation on graphs
I sort of understand when do I use variational Bayesian and when do I use expectation maximization. But now I want to know when do I use belief propagation in graphs to solve an estimation problem.
...
2
votes
1answer
51 views
EM Algorithm For Bipolar Normal Distribution
Question:
Let $x_1, \dots, x_m$ be an i.i.d. sample from a normal density with mean $\mu$ and variance $\sigma^2$.
Suppose for each $x_i$ we observe $y_i = |x_i|$ . Formulate an EM algorithm for
...
2
votes
1answer
47 views
Nonparametric mixture estimation
Let's assume that we have two samples $\{X_i\}_{i=1..N}$ and $\{Y_i\}_{i=1..M}$ corresponding to random variables $X$ and $Y$. Let there also be a sample $\{Z_i\}_{i=1..K}$ corresponding to random ...
1
vote
0answers
80 views
Calculating ELBO in EM algorithm
In Andrew Ng's CS229 notes, a nice derivation of EM algorithm is given. With some minor notation modifications, the algorithm is written as follows:
E-step: For each i, $Q_i(z_i)=p(z_i|x_i, \theta^{...
6
votes
1answer
128 views
Finding the Q function for the EM algorithm
I have a situation where $X_1,...X_n$ come from $N(\mu,1)$ and there is a realization of 10 $x$ values. I want to use the EM algorithm to work out the MLE.
So, I am trying to compute the expected ...
2
votes
1answer
64 views
Why maximizing the expected value of log likelihood under the posterior distribution of latent variables maximize the observed data log-likelihood?
I am trying to understand the Expectation-Maximization algorithm and I am not able to get the intuition of a particular step. I am able to verify the mathematical derivation but I want to understand ...
0
votes
0answers
20 views
Fixing the parameters of the variational distribution in Expectation-Maximization
Consider directed graphical model $z \to x$ (with $z$ unobserved and $x$ observed). The evidence lower bound on the log-likelihood $\log p(x) = \log \sum_z p(x, z; \theta)$ for parameters $\theta$ (...
1
vote
1answer
84 views
How to get number of iterations in EM-algorithm using R mclust gaussian mixture model
I am clustering data using the mclust function from the R mclust package. I am struggling to get the number of iterations the EM ...
2
votes
1answer
55 views
derivation of E step in EM algorithm for pLSA via Lagrangian
I have trouble deriving the EM algorithm for the Probabilistic latent semantic analysis (pLSA) model via Lagrange multipliers.
I model the missing data $Q_{zij} \in \{0,1\}$ for word $w_j$ in document ...
1
vote
2answers
47 views
Understanding Bishop on EM for HMM's
I'm reading page 616 of Bishop's PRML (pdf), which introduces EM for hidden markov models with categorical hidden states and arbitrary emission distributions. Bishop defines $z_n$ as the hidden state ...
0
votes
0answers
9 views
EM, MCMC and VB, What are their similarity, difference and use scenorio?
I am recently learning MCMC, and realized some connections between MCMC, EM and VB (variational Bayesian). However, I did not find any discussion that can clear my mind on their similarity, difference ...
1
vote
1answer
43 views
Are EM and MCMC closed-form analytical solution
I came across the concept of closed-form analytical solution while learning MCMC and got a bit confused. At this point, my understanding is that the problem does not have closed-form solution if it ...
0
votes
0answers
16 views
The performance of EM algorithm at different local maximas
Setup: Say I have a problem (any problem that can be modeled by a probabilistic model) to solve, this problem involves some unknown parameters to estimate. I model this problem in a probabilistic ...
4
votes
2answers
332 views
What is the relationship between VAE and EM algorithm?
What's the relationship between Variational Autoencoders and the Expectation Maximization Algorithm?
I know that the EM algorithm is used in latent variable models, specifically to do maximum ...
0
votes
0answers
25 views
Solving Markov Switching Autoregressiv (MS-AR) model
I am currently trying to estimate a MS-AR model of the form
$$y_t - \mu_{s_t} = \sum_{l=1}^{p}\phi_l(y_{t-l}-\mu_{s_{t-l}}) + \epsilon_t$$
with $p=4$ and $n=3$ regimes. Further, the variance terms are ...
2
votes
1answer
120 views
EM Algorithm Derivation, Discrete Case
Just wanted to ask whether the following derivation is correct:
Suppose $X$ is a vector of observed random variables, $Z$ is a vector of unobserved random variables and $\theta$ is a vector of ...
2
votes
0answers
21 views
Multi-layer regression (with non-linearity in between): Iterated regression algorithm?
Suppose we observe $z_1,\dots,z_n\in\mathbb{R}$ and $x_1,\dots,x_n\in\mathbb{R}^b$ generated from the following model, for $i=1,\dots,n$:
$$y_i=x_i^\top \beta,$$
$$z_i={f(y_i)}^\top \gamma + \...
0
votes
0answers
17 views
What assumption need to hold for Monte Carlo EM algorithm?
I have a hierarchical(/multilevel) model from which I want to estimate its parameters. I do by the use of Maximimum Likelihood Estimation via the (Markov Chain) Monte Carlo EM algorithm. When solving ...
4
votes
1answer
284 views
Derivation of M step for Gaussian mixture model
Summary
So to summarize my question, how can I take
\begin{align}
= \sum_{i=1}^{n}W_{i1} \left(log (1-\sum_{j=2}^{K}\pi_j) -\frac{1}{2} log(|\Sigma_1|) -\frac{d}{2} log(2\pi) -\frac{1}{2}(x_i-\mu_1)^{...
0
votes
0answers
28 views
Gaussian mixture model parameter updates derivation?
I've been following a helpful tutorial and I'm trying to understand the parameter updates.
For example, the mu_k parameter update is below. I'm unsure why the sum(Bk) does not cancel out as it's in ...
3
votes
0answers
31 views
Expectation-Maximisation derivations [duplicate]
I've come across a few different sources on expectation-maximisation which I can't quite match up.
The CS229 lecture 8 [1] states that the function we must write down and maximise is:
$$
Q_1 = \sum_{...
0
votes
0answers
11 views
Is this the correct expression for variational distribution q(t) for expectation maximization algorithm?
Just hoping for a sanity check on my equations below.
Ultimate goal: apply the EM algorithm to a GMM problem. First I need to come up with an expression for
the variational distribution q(t) where ...
1
vote
1answer
31 views
The connection between the expectation in expectation maximization and the importance sampling?
The log-likelihood of the EM algorithm can be expressed as
\begin{align}
\ell(\theta, x) &= \log p(x|\theta) \\
&= \log \sum_z p(x, z|\theta) \\
&= \log \sum_z \frac{q(z|x)}{q(z|x)}p(x,z|...
0
votes
0answers
14 views
Fitting a mixture model distribution to kurtotic data
I need to fit a parametric distribution to data that has non-zero (unknown) kurtosis. First I tried to fit a Pearson type VII / Student's t, but the fitting is especially poor in the two tails, ...
0
votes
1answer
22 views
Nonlinear filtering with unknown model parameters
I have a Markov chain with scalar states $x_t$ evolving according to
$$x_t = x_{t-1} + \boldsymbol\vartheta^{\rm T} {\boldsymbol\varphi_{t-1}}(x_{t-1}) + w_t$$
with unknown but constant parameters $\...
0
votes
1answer
57 views
M step EM algorithm in Mixture Models. Expected value of the indicator variable under the posterior [closed]
I am not able to solve the following expectation. In the EM algorithm, the first step in the M step is to compute the expected value of $\log p(x,z)$ where $x$ are observations and $z$ indicator ...
1
vote
1answer
112 views
Why marginal likelihood is optimized in expectation maximization?
Suppose we would like maximize a likelihood function $p(\mathbf x, \mathbf z| \theta)$, where $\mathbf x$ is observed, $\mathbf z$ is a latent variable, and $\theta$ is the collection of model ...
1
vote
1answer
51 views
MCMC inside Expectation Maximization
I wish to optimize the following likelihood function for parameter $\Theta$:
$$p(D|\Theta)=\int_X\int_Y p(x, y, D|\Theta)dydx$$
where $X$ and $Y$ are latent variables and only $D$ is observed. I ...
2
votes
1answer
81 views
Exponential Reestimation Formula in EM Algorithm
I'm trying to understand how to reestimate parameters, as part of the EM algorithm. As a simple example, I'm trying to derive the reestimation formula for an exponential distribution. Here's the setup:...
