Questions tagged [expectation-maximization]
An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.
0
votes
1answer
10 views
Consistency between EM clusterings with varying starting point
I have a data set (~9 dimensions) in Weka and am running the EM clusterer with a fixed number of clusters. When changing the seed/initial point, the clusterings are very different. Is this expected?
...
0
votes
0answers
16 views
EM algorithm for factor analysis,the formula of diagonal matrix stuck
I am trying to learn the factor analysis of CS229,the relative lecture note is here:CS229 Lecture note9
I have stucked at the diagonal matrix formula which is at the page 9:
What's the derivation of ...
0
votes
0answers
44 views
EM Derivation for Dawid-Skene Model
I am trying to derive the EM update equations for the Dawid-Skene model. Following the notation in Bayesian Classifier Combination by Kim and Ghahramani, $i$ is the index of the data point, $t_i$ is ...
0
votes
0answers
13 views
WEKA: splitting training and testing data and using PCA feature transformation and clustering for Neural Network training
I'm working on an assignment that requires me to split some data into a training and testing set, learn a Principal Component Analysis feature transformation on the training set and project it onto ...
0
votes
0answers
13 views
Applying Expectation Maximization Algorithm to Mixture of Exponential Distributions
I am new to EM algorithm and I'm dealing with the following question of a Mixture of Exponential Distributions.
Suppose that the time from when a machine is manufactured to when it fails is ...
0
votes
0answers
9 views
Linear regression of features inside a hidden Markov model?
I have an interesting little problem which I am trying to attack using HMMs.
First, as usual, I am trying to do time-series segmentation/classification using a HMM. But the input to my HMM has an ...
1
vote
1answer
47 views
Estimating truth and confusion matrix from noisy observations with Expectation Maximization?
Suppose we have $m$ sources, each of which noisily observe the same set of $n$ independent events from the outcome set $\{A,B,C\}$. Each source has a confusion matrix, for example for source $i$:
$$...
1
vote
0answers
58 views
how can I do online Expectation–maximization algorithm?
em algorithm is usually optimized iteratively between the expectation (construct a lower-bound) and the maximize the likelihood (optimize the lower-bound) to guarantee convergence. However, at each ...
0
votes
0answers
23 views
MLE for Mixture Model [duplicate]
$$P(W) = \sum_{k=1}^K \pi_k P(W|\mu_k)$$
$$P(W|\mu_k) = \prod_{i=1}^M(\mu_k(i))^{W(i)}$$
Here, it is a mixture model and $W$ is a one-hot encoded vector over a dictionary of size $M$ ie $\sum_{i=1}^{...
0
votes
0answers
30 views
EM algorithm and AIC criteria
I am using EM algorithm to estimate the model parameters. EM-algorithm iterates until the loglikelihood is converged. After that, I need to compute AIC criteria. As known, AIC is a loglikelihood ...
1
vote
2answers
128 views
Comparing K-Means and Expectation Maximization on the dataset generated - When does K-Means perform better?
I was experimenting with K-Means and Gaussian Mixture Models (Expectation-Maximization) on the data set that I generated. Here is how the plot for two distributions looks like:
Since this was ...
1
vote
0answers
18 views
Likelihood Construction for Censored Data
I am trying to understand the Expectation-Maximization algorithm, and was trying to read through this paper by Park and Lee. In section 2, "Likelihood Construction for Censored Data", they mention the ...
2
votes
2answers
233 views
Question about the latent variable in EM algorithm
In mixture models, Expectation maximization algorithm (EM) is a commonly used method to estimate the model parameters. Suppose that I have bivariate mixture model with two mixture components, with ...
1
vote
1answer
34 views
What is an Expectation Maximisation Algorithm for Markov chains?
I'm looking for an algorithm for Expectation Maximisation of a Markov chain.
I am aware of the Baum-Welch algorithm for Hidden Markov Models, but I can't find an algorithm for Markov Models that are ...
0
votes
1answer
35 views
The ways to normalize the likelihood in EM algorithm
In Wikepedia it states that:
In the simplest cases, normalization of ratings means adjusting values
measured on different scales to a notionally common scale, often prior
to averaging.
And ...
0
votes
0answers
25 views
EM algorithm early stopping
Assume there are a set of latent variables $X$, a set of observed variables $Y$ and some parameters $\Theta$. I am using EM algorithm to compute $X$ and $\theta$. In the E step, it computes $p(X|Y,\...
2
votes
0answers
29 views
Is Expectation-Maximization the right type of analysis (& if so, how)?
I have 118 microbes that I have tested at various concentrations of a drug that is supposed to kill them (0.1875 - 15). However, these microbes have mutations (A-H) which confer resistance to the ...
1
vote
0answers
40 views
Optimal Sequence Problem
Lets says I have 100 people that like to buy item x. They ask me to send them a message every time I have x available to sell.
Of the 100 people that like to buy item x:
1) Some people will pay more ...
1
vote
0answers
47 views
MLE for high dimensional $\theta$
I'm estimating a parameter $\theta$ in the context of covariance structure model given by $\Sigma(\theta)$. As an estimator, I use ML and computation is done by fmincon function in Matlab(using sqp ...
4
votes
0answers
154 views
Fisher information matrix in logistic regression
I am self-studying the basics of logistic regression. I came across this sentence:
In logistic regression expected and observed information matrixes are
equal
I am aware that the information ...
1
vote
1answer
14 views
In the expectation step, why do we sometimes assign the data to a component (i.e. complete the data) instead of calculating the expected value?
Let $Y|X$ be a mixture distribution conditional on covariates $X$, with distribution function $Y(x; \sigma, \psi, \phi) = \alpha Y_1(x; \psi) + (1 - \alpha) Y_2(x; \phi)$, for the averaging parameter $...
0
votes
0answers
15 views
How to use the estimate given by the EM algorithm to guess at the missing value
Text: Computational Statistics 2E by Givens and Hoeting
Example 4.1: Simple Exponential Density
The set-up is as follows: Suppose that $Y_1, Y_2 \overset{\text{iid}}{\sim} \rm{Exp}(\theta)$ and that ...
2
votes
1answer
168 views
Gaussian Mixture Model
with the following code I fit a Gaussian Mixture Model to arbitrarily created data. The code is working. The only thing I encounter is that during the calculation of the multivariate_normal I ...
2
votes
0answers
18 views
Multivariate mixture models
I am new to mixture modeling and have successfully used bernoulli mixture models to cluster datasets of binary data.
My real purpose, though is to cluster datasets with mixed data types: normal, ...
0
votes
0answers
29 views
Sample covariance matrix notation
I do not understand this notation for the sample covariance matrix (from Artificial Intelligence: A Modern Approach, Peter Norvig and Stuart J. Russell, Section 20.3, EM algorithm):
$\Sigma_{i} = \...
0
votes
0answers
57 views
Likelihood scaling for Bernoulli Mixture Model to avoid underflow
I am very new to Expectation Maximization and struggling with how to scale the likelihood calculations to avoid numeric underflow.
I am trying to create a Bernoulli Mixture Model for sparse, binary ...
0
votes
0answers
16 views
Confusion over expectation maximisation algorithm
first of all, apologies if this is the wrong place for this.
I've been reading around my actuarial studies, and came across the expectation maximisation algorithm. I first read this
article, which ...
2
votes
1answer
77 views
Why is the prior omitted from this Bayes rule?
I'm trying to understand the EM algorithm. I've found a tutorial on it. It goes like this:
Two coins (A & B). 5 rounds of flipping 10 times. We forgot, however, which coin was flipped each round. ...
3
votes
0answers
38 views
Termination Condition(s) for Expectation Maximization
What are good criteria for deciding when to terminate the expectation-maximization algorithm.
I know that the idea is that you should terminate when the change in the data log likelihood is "small" ...
0
votes
1answer
39 views
Usefulness of EM algorithm
I wonder how EM make things easier when we are finding the MLE with missing data.
Let $Z$ be the complete data, $Y = Y(Z)$ the observed data, and $\theta$ the parameter to be estimated.
For the MLE, ...
4
votes
1answer
494 views
Is this a typo/error in Bishop's book
I am currently going through the chapter 9 - Mixture Models and EM from Bishop's book - Pattern Recognition and Machine Learning (2006).
I could not understand the maximization step with respect to ...
1
vote
1answer
38 views
Distribution function or density in Mixed Distribution EM
In this calculation
https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm#E_step
a probability, $P(Z_i = j|X_i=x_i;\theta^{(t)})$ is evaluated using bayes theorem, and then each ...
2
votes
0answers
104 views
Using Naive Bayes classifier for unsupervised learning
I was going through this article to learn about how the EM algorithm can be used to use the Naive Bayes algorithm for unsupervised learning. Suppose we have the following data without labels:
1 0 1 1
...
2
votes
1answer
76 views
What are some applications of unsupervised HMMs?
Supervised HMMs can be applied to many problems like POS tagging and OCR (optical character recognition).
I've learned that HMMs can be trained unsupervisedly using EM (Baum-Welch algorithm), what ...
1
vote
1answer
419 views
Estimating truncation point in Gaussian mixture
I have data modeled as a mixture of two Gaussian distributions. The data is "clipped" i.e., there is data only for values greater than a threshold $t$, even though it is feasible for data to exist in ...
0
votes
2answers
52 views
Why different initial parameters of Expectation–Maximization (EM) result in different clusters? [closed]
I'm having a hard time understanding conceptually why we get different clusters when we start the algorithm with different initial parameters. Can anyone explain the mechanisms behind it to me a ...
1
vote
2answers
164 views
Why the EM algorithm instead of a more direct computation? [duplicate]
Assume we have a probability distribution $P(y,z|\theta)$,
where $x$ is the total set of variables divided into observable variables $y$
and hidden variables $z$,
and data on observable variables $y$.
...
1
vote
1answer
188 views
Covariance matrix of image data is not positive definite matrix
I've really hit the wall here and need help with direction :).
I am trying to use mvnpdf as part of basic EM algorithm but the covariance matrix of data seems to be not positive definite. There are ...
1
vote
1answer
217 views
EM Algorithm seems to work, but Q is not monotonic. Possible reasons?
I have implemented Expectation maximization to fit some of the parameters of a linear Gaussian state space model using Kalman filtering / smoothing.
The model is:
$x(t) = Ax(t - 1) + w(t); w(t) \sim ...
1
vote
0answers
3 views
How to estimate the observation level data generating when there is unequal sampling of data for each record
I am looking at bid amounts for cars sold at a sealed auction (only the seller has all the information). I am trying to predict the number and magnitude of offers that a car will receive. Some of the ...
0
votes
0answers
23 views
EM algorithm coupled latent variables
Assume we have a set of latent variables $X = \{x_1,x_2,\dots,x_n\}$, a set of model parameters $\theta = \{\theta_1,\theta_2,\dots,\theta_m\}$ and a set of observed variables $Y= \{y_1,y_2,\dots,y_n\}...
4
votes
1answer
159 views
How to picture EM algorithm and KL-divergence geometrically?
In reading up on the Expectation-Mmaximization algorithm on Wikipedia, I read this short and intriguing line, under the subheading "Geometric Intuition":
In information geometry, the E step and the ...
0
votes
0answers
78 views
Estimate parameters from truncated normal sample [duplicate]
I have a question like this, $X \sim N(\mu,\sigma^2)$ with unknown parameters. Now, a sample of size $m$ generated from X, but filter by X < T, i.e., any number larger than T will be ignored and ...
2
votes
0answers
185 views
GMM EM algorithm complexity per iteration
I was fitting GMM clusters with diagonal covariance on my data using EM with $n$ (=5e6) points, each having $m$ (=160) ...
1
vote
1answer
181 views
EM algorithm with constraints
Lets say that we want to constraint some of the variables in a model, we denote the model's parameters by $\theta \in \mathbb{R}^n$ and we want to train the model on input data $X$.
Normally we would ...
0
votes
0answers
26 views
Conditional expectations of the missing values in EM
I'm trying to figure out the EM algorithm for the right-censored linear regression. In order to perform the M step I need to find the conditional expectations of the missing values:
My question is ...
0
votes
2answers
110 views
latent dirichlet allocation: complexity and implementation details
I was confused by how LDA (by the original variational inference) can be implemented in a way such that the number of operations for each document $j$ is $\mathcal{O}(N_j~K)$, where $N_j$ is the ...
0
votes
0answers
21 views
Notation of “integrate over the unobserved data” in Expectation-Maximization
In the context of Expectation-Maximization, I have always written:
$$
p(y |\phi) = \int_z p(y, z | \phi) dz
$$
Reading through some theoretical papers, though, I find this notation
\begin{align}
g(y |...
2
votes
2answers
142 views
This is Gaussian mixture model?
Here is a problem that I am looking at.
Is this model really commonly known as a Gaussian mixture model (the one often appears as an illustration of EM algorithm)?
I am confused because Gaussian ...
1
vote
2answers
173 views
Variational inference: how to rewrite ELBO?
I am reading this paper on variational inference and this website.
One thing I am confused about is how they get to decompose ELBO, where $ELBO(q) = E_q[log~p(z,x)] - E_q[log~q(z)]$, when focusing ...
