Questions tagged [expectation-maximization]
An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.
491
questions
3
votes
0answers
28 views
Expectation-Maximisation derivations [duplicate]
I've come across a few different sources on expectation-maximisation which I can't quite match up.
The CS229 lecture 8 [1] states that the function we must write down and maximise is:
$$
Q_1 = \sum_{...
0
votes
0answers
11 views
Is this the correct expression for variational distribution q(t) for expectation maximization algorithm?
Just hoping for a sanity check on my equations below.
Ultimate goal: apply the EM algorithm to a GMM problem. First I need to come up with an expression for
the variational distribution q(t) where ...
0
votes
1answer
22 views
The connection between the expectation in expectation maximization and the importance sampling?
The log-likelihood of the EM algorithm can be expressed as
\begin{align}
\ell(\theta, x) &= \log p(x|\theta) \\
&= \log \sum_z p(x, z|\theta) \\
&= \log \sum_z \frac{q(z|x)}{q(z|x)}p(x,z|...
0
votes
0answers
11 views
Fitting a mixture model distribution to kurtotic data
I need to fit a parametric distribution to data that has non-zero (unknown) kurtosis. First I tried to fit a Pearson type VII / Student's t, but the fitting is especially poor in the two tails, ...
0
votes
1answer
17 views
Nonlinear filtering with unknown model parameters
I have a Markov chain with scalar states $x_t$ evolving according to
$$x_t = x_{t-1} + \boldsymbol\vartheta^{\rm T} {\boldsymbol\varphi_{t-1}}(x_{t-1}) + w_t$$
with unknown but constant parameters $\...
0
votes
1answer
36 views
M step EM algorithm in Mixture Models. Expected value of the indicator variable under the posterior [closed]
I am not able to solve the following expectation. In the EM algorithm, the first step in the M step is to compute the expected value of $\log p(x,z)$ where $x$ are observations and $z$ indicator ...
1
vote
1answer
42 views
Why marginal likelihood is optimized in expectation maximization?
Suppose we would like maximize a likelihood function $p(\mathbf x, \mathbf z| \theta)$, where $\mathbf x$ is observed, $\mathbf z$ is a latent variable, and $\theta$ is the collection of model ...
1
vote
1answer
32 views
MCMC inside Expectation Maximization
I wish to optimize the following likelihood function for parameter $\Theta$:
$$p(D|\Theta)=\int_X\int_Y p(x, y, D|\Theta)dydx$$
where $X$ and $Y$ are latent variables and only $D$ is observed. I ...
2
votes
1answer
63 views
Exponential Reestimation Formula in EM Algorithm
I'm trying to understand how to reestimate parameters, as part of the EM algorithm. As a simple example, I'm trying to derive the reestimation formula for an exponential distribution. Here's the setup:...
-1
votes
1answer
35 views
How to fit a mixture of a normal and a half normal distribution?
I tried Expectation-Maximization (EM) based fitting using the mixfit function from the mixR package in the R environment. It yielded a normal mixture model with 2 components:
1) pi 0.21, mu: 0.47, sd: ...
0
votes
0answers
15 views
Estimate parameter estimates of hierarchical model
I have the following hierarchical model,
where t stands for time and y_t, x_1t, x_2t, ..., z_1t, z_2t,... are known. I want to estimate the parameters of the model using a frequentist approach, not a ...
0
votes
0answers
10 views
Compute membership probabilities in E-step of EM algorithm with log-densities instead of densities
As an exercise I have implemented the EM algorithm for Gaussian mixtures, however, I have the problem that in high dimensions the densities of data points become so small that I get a numerical ...
0
votes
0answers
24 views
Clarification of a passage of Bishop about EM algorithm
I am trying to get a very good grasp of the EM algorithm, including the MCEM variant. There is one little passage of the famous Bishop book (pag 440, Mixture Models and EM) where it says:
Now ...
2
votes
1answer
36 views
Derivation of EM in bishop
I'm working through chapter 9 in Bishop (Mixture models and EM) and I'm stuck on equation 9.29. For those without the book:
Bishop states that the log likelihood for a latent variable model with
...
1
vote
0answers
28 views
Random forest (or regression) when predictors are intervals
I'm trying to fit a Random Forest model where the predictors are reported as an interval rather than a point estimate. The structure of each data point is a triplet $(\bar{y_i}, x_{i1}, x_{i2})$ where ...
0
votes
0answers
19 views
Evaluating goodness of fit of a model estimated with EM-algorithm (with AIC or BIC)
I am learning a Hidden Markov Model with time varying transition probabilities depending on different features. I do this by estimating the model parameters with the EM-algorithm. Now I would like to ...
0
votes
0answers
40 views
E-step of E-M algorithm with missing data
I am learning expectation-maximization (E-M) algorithm on Coursera and during the course the teacher says that it can be used to handle missing data when fitting Gaussian mixtures (GM) but did not ...
0
votes
0answers
48 views
EM algorithm for mixture of categorical distributions instantly stabilizes
Brief Summary of Question
I'm trying to fit a mixture model of categorical distributions (see https://en.wikipedia.org/wiki/Categorical_distribution). The expectation at the second time step is ...
1
vote
1answer
56 views
Use Expectation-Maximization algorithm for obtaining maximal likelihood estimator
For $X = {(Z_{i}, Y_{i}) : i = 1, ... ,n}$, consider the model:
$Y_{i} = \beta_{1} + \beta_{2}Z_{i} + \epsilon_{i}$
where $\epsilon_{1}, ... ,\epsilon_{n}$ are i.i.d $N(0,\sigma^2)$, $Z_{i},...Z_{i}$ ...
0
votes
0answers
11 views
Applying the EM algorithm for model with 'fixed' coefficients
My problem is that I want to apply the EM algorithm on a stochastic model (knowing full well that the model is misspecified).
\begin{align} log(y_{t}^2) &= h_t + log(\epsilon_t ^2) \\
h_{t+1} = \...
4
votes
1answer
81 views
Derivation of maximum likelihood for a Gaussian mixture model
I'm working my way through the derivation of EM in Bishop (p. 435).
I'm stuck trying to derive to MLE for $\mu_k$ for the gaussian mixture model.
Basically I get an extra sum in the numerator.
For ...
0
votes
0answers
15 views
Bayesian PCA, problem understanding Expectation Maximization scheme
I'm reading the following article https://papers.nips.cc/paper/1549-bayesian-pca.pdf of Christophe M. Bishop. I've understood the general method, however, I have trouble understanding the EM scheme ...
2
votes
1answer
38 views
Why don't we treat the mean and variances in EM algorithm as latent variables
I know how the Expectation Maximization works. What I fail to understand is why only the mixture components are treated as latent variables and why not the mean and variances values of the K gaussians?...
0
votes
1answer
25 views
What is the Q distribution in expectation maximization in the following explanation?
I am reading a blog on expectation maximization - http://krasserm.github.io/2019/11/21/latent-variable-models-part-1/
Here, I encounter the following expression:
When you look at the above ...
0
votes
1answer
18 views
In expectation maximization, why do we have a latent variable distribution for every sample of the data
I am reading this blog on expectation maximization - http://krasserm.github.io/2019/11/21/latent-variable-models-part-1/
Starting the section where the author starts explaining how EM is done in the ...
5
votes
0answers
99 views
Does EM algorithm require us to know the joint (predictive) distribution of the latent variables $Z$ when $Z$ is two-dimensional?
In its general form the E-step of the EM algorithm finds the expectation
$$ Q(\theta|\theta') =\int \log[ p(Y,Z | \theta)] p(Z|Y,\theta') d Z$$
where $Y$ the data, $Z$ the latent variables, $\theta'$...
4
votes
1answer
150 views
ELBO maximization with SGD
In cases such as Gaussian mixture models, there's is no closed-term solution for the original likelihood maximization. Maximizing the ELBO, however, does have analytical update formulas (i.e. formulas ...
2
votes
2answers
87 views
interpretation of the estimated parameters of a gaussian mixture model
I need to find/fit a model for the color of an object. Suppose its color is generally yellow and we have 10000-by-3 data which are pixel values for R, G, B channels.
Firstly I choose a Multivariate ...
1
vote
1answer
27 views
Gaussian Mixture model - Penalized log-likelihood in EM algorithm not monotone increasing
I am working on a multivariate Gaussian Mixture Model in R. The goal is to do regularized clustering on the data, where each component represents a cluster.
I wrote an EM algorithm to maximize a ...
0
votes
0answers
20 views
How to calculate weights in EM-algorithm?
Assume that we have 2 clusters with some initial model:
Can you please explain how weights are calculated?
1
vote
1answer
15 views
Fitting mixture model on data with duplicate values
What is the correct procedure to fit finite mixture models on data with many duplicate values using EM? Let's say I have N(0,1) distributed data and try to fit a 2 component mixture using EM. There ...
1
vote
0answers
20 views
Confusion in Sampling using the IP algorithm (Bishop PRML)
I'm reading Bishop's PRML p. 537 and I don't understand one piece of the IP (data augmentation) algorithm. Namely, the part that says "we use the samples $\{\mathbf{Z}^{(l)}\}$ obtained in the I step ...
0
votes
0answers
31 views
Newton-Raphson method to solve for dof when performing MLE of a multivariate Student-t distribution using EM
I am reading the derivation of EM algorithm to estimate the maximum likelihood of a multivariate Student-t distribution $\mathcal{T}(\mathbf{x} \vert \pmb{\mu}, \pmb{\Sigma}, \nu)$ in Kevin Murphy's ...
0
votes
0answers
42 views
Why Expectation and Maximization algorithm not used in Machine Learning while Gradient Descent algorithm used in Machine Learning?
I know that Newton Raphson, Expectation & Maximization, and Gradient Descent are all known to be optimization methods. Somehow, I wonder why Gradient Descent is chosen to be used in most of ...
0
votes
0answers
27 views
Do I impute missing values with the response?
I have a dataset with missing values in both predictors and the response. As far as I know, the data are missing not at random, so I cannot simply use listwise deletion. Instead, I employed the EM ...
2
votes
0answers
65 views
Can someone verify if the following Bayesian Information Criterion (BIC) model selection algorithm is correct for Gaussian mixture models?
I am trying to find an automated way of picking the number of clusters $K \in \mathbb{N}$ for unsupervised learning scenarios, specifically for GMM.
I was suggested to use something called the "...
4
votes
0answers
41 views
Expectation Maximisation (EM) Algorithm
Some of my parameters do not have a closed form solution. Thus, for these parameters the M-step is implemented via a one-step Newton-Raphson update, i.e.,
\begin{equation}
\theta^{t+1} = \theta^t - \...
0
votes
0answers
15 views
Outlier detection with EM
I am interested in using expectation maximization for outlier detection. In the literature this is usually done assuming that the data of interest are normally distributed while the outliers are ...
0
votes
0answers
15 views
Using expectation maximization for robust regression
What are the advantages/disadvantages of using EM for robust estimation vs. the robust estimation with Huber or Tukey loss functions?
0
votes
0answers
34 views
assessing the stability of importance (sampling) weights
I have read that when importance weights are used, the stability (variability) of the weights should be assessed (Levine and Casella, 2001) -- however, I wonder how this might be accomplished.
For ...
2
votes
2answers
53 views
EM-algorithm for two clusters (when one of the distributions is uniform)
I am having a hard time with the EM-algorithm. Here's the problem that I am trying to solve.
Dealing with noisy annotations is a common problem in computer vision, especially when using ...
3
votes
2answers
81 views
Accounting for uncertain information (few observations) in a prior (empirial Bayes)
I did not really know how to choose an adequate title for this question, so please feel free to change it.
I have a weird case wherein frequentist and Bayesian philosophies come together. I am ...
1
vote
0answers
42 views
Expectation Maximisation vs Expectation Propagation in the context of Bayesian Networks
I am confused about Expectation Maximisation and Expectation Propagation algorithms in the context of Bayesian Networks, especially whether one comprise another.
What is the difference between ...
1
vote
0answers
9 views
Consistency of EM for missing data in non-parametric setting
When we have missing data, a parametric model, and an expectation-maximization procedure, and we want to show that our procedure leads to consistent estimators, we can sometimes set up score functions ...
1
vote
1answer
156 views
Calculation of AIC in finite mixture modeling
I have a question about calculation the AIC to find my optimal amount of clusters. I am applying mixture modeling with the EM algorithm. I know the formula AIC = -2ln(log-lik) + 2k.
These are my log-...
0
votes
0answers
21 views
Analysing faithful dataset in R using GMM
I have got a project on analysing the faithful data in R found in the package "datasets" and called using data(faithful) which is the data set off eruption time and waiting time of the Old Faithful ...
1
vote
0answers
53 views
Are my data a good candidate for EM imputation followed by exploratory factor analysis?
I am doing Exploratory Factor Analysis (EFA) in R, using principal axis factoring in the psych package. I have missing data that prevent me getting factor scores, so I am imputing data. I am using ...
0
votes
0answers
21 views
Cosine Similarity for Classification to EM Cluster?
Perhaps my question sounds naive, uncovering the very little knowledge that I have in the field of Statistics, but is very urgent to get a solid answer or trigger for further insights for my concerns.
...
1
vote
0answers
40 views
A trivial question about EM algorithm theory
In "The EM Algorithm and Extensions", second edition, from Geoffrey J. McLachlan and Thriyambakam Krishnan, X is the latent variable, and Y is de observed (incomplete) variable
I'm little confuse ...
1
vote
1answer
116 views
Using AIC/BIC within cross-validation for likelihood based loss functions
For a course I am teaching, I am having my students fit a Gaussian mixture model using MLEs via the EM algorithm to a bivariate dataset. I have asked the students to use use cross-validation to choose ...
