An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.

learn more… | top users | synonyms (1)

0
votes
0answers
6 views

Fast soft-assign to a large high-dimensional GMM

We wish to perform a fast (possibly approximate) soft-assign of high-dimensional data points to a large Gaussian mixture with diagonal covariance (this is related to the E step of the EM algorithm). ...
2
votes
0answers
19 views

EM algorithm: With prior vs. not prior

I have a working EM algorithm without prior. I am asking for some advice on how to add prior on latent variables. Define: $t_i \in \{ +1, -1 \} $: variables of interest to be predicted $p_j \in ...
4
votes
1answer
86 views
+50

What will be the estimator for these parameters

Question: $y_0 = z^d$ is computed from the sum of some recordings by a sensor. Let, there be $k$ sensor nodes. This parameter is calculated by each sensor node and then transmitted to the base ...
0
votes
0answers
7 views

Strange representation of the loglikelihood of the mixture model with truncated data - EM Algorithm

Assume I have a mixture model $f(x) = \sum_{k=1}^{m}\alpha_k f_k(x)$. For this density, we must have the survival function $\bar{F}(x) = \sum_{k=1}^{m}\alpha_k \bar{F}_k(x)$ Assume now that I ...
1
vote
3answers
29 views

When does the EM for Gaussian mixture model has one of the Gaussian diminish to exactly one point and have zero variance?

I had implemented the EM algorithm for mixture models as follows: For the E-step I compute the soft-counts of assigning each point $x^{(t)} \in Data_n$ to an individual cluster $j \in \{1, ..., K \}$ ...
1
vote
0answers
30 views

How do you do EM algorithm for a factored model for a recommender system?

Let $X$ be a $n \times d$ matrix with users as rows and movies as columns. Each user is a single row $x^{(u)} \in \mathbb{R}^d$ (i.e. for user u there are at most d ratings for the d movies). Also ...
1
vote
0answers
22 views

EM algorithm for left truncated right censored (life time) data

In the general framework of EM algorithm applied to statistical inference of mixture distributions/models, you have a set of latent (unobservable) variables $z_{ij}$ denoting if observation $i$ is an ...
4
votes
1answer
109 views

Convergence from the EM Algorithm with bivariate mixture distribution

I have a mixture model which I want to find the maximum likelihood estimator of given a set of data $x$ and a set of partially observed data $z$. I have implemented both the E-step (calculating the ...
2
votes
1answer
44 views

Expectation Maximisation

I'm currently reading Thomas Hofmamms paper on Probabilistic Latent Semantic Analysis. He includes a formula for the E step in Expectation Maximisation, but has proposed an alternative to this step ...
1
vote
1answer
46 views

Kalman Filter Expectation Maximization

I'm not very familiar with the EM algorithm for the Kalman Filter. I've been using pykalman to do my analysis in Python. The package comes with a simple EM algo: ...
0
votes
1answer
22 views

Convergence of EM in Mixture Models w.r.t unlikely events $(f(\cdot)=0)$ in either distribution

To maximize the likelihood of a mixture model with unobserved latent variables, the Expectation Maximization is conventionally applied. Assuming we have data $x_1,\dots,x_n$ from a fixed number of ...
1
vote
0answers
15 views

Non-Causal time-series filtering techniques for standard noise with unkown variance. (EM vs. weiner vs. kalman)

This is a quick question about filtering stored time-series data using kalman/weiner filtering techniques or expectation maximization. I'm just hoping to fix some confusion about questioning ...
0
votes
0answers
19 views

How to evaluate the goodness of Fit of parameters obtained from EM algorithm

I have a set of observations $\mathcal{Y} = {Y_1, \cdots, Y_T}$. I am running EM algorithm to fit the observations to the following Hidden Markov Model $$A = [a_{ij}]_{N \times N}, a_{ij} = P(X_{k+1} ...
2
votes
0answers
30 views

Beginner level: How to plug in the smoothing equations into E step (Part 2)

Considering Gaussian Linear Dynamical system, $x_{t+1} = Ax_t + w_t$ $y_t = Cx_t + v_t$ $w_t = N(0,Q)$, $v_t = N(0,R)$ By Kalman Filter we are estimating the state variables and the state estimate ...
1
vote
1answer
168 views

Why is there a E in the name EM algorithm?

I understand where the E step happens in the algorithm (as explicated in the math section below). In my mind, the key ingenuity of the algorithm is the use of the Jensen's inequality to create a lower ...
4
votes
2answers
67 views

Expectation maxmisation algorithm increases true likelihood at each iteration

I've heard that the EM algorithm ensures that the true likelihood is non-decreasing at each iteration of the algorithm, but I'm not sure why this is the case. I've provided a basic plot which I ...
0
votes
0answers
58 views

Maximum likelihood estimate parameters estimation

In this tutorial on mixture models, page 2, how did the author arrive to the parameters for maximum likelihood in the fully observed case? This is the general setting (based on an excerpt from the ...
0
votes
0answers
17 views

Fitting multiple power laws, Zipf's law in the real-world

As a preface, the following questions are related: How to calculate Zipf's law coefficient from a set of top frequencies? How to estimate parameters for Zipf truncated distribution from a data ...
0
votes
0answers
18 views

pLSA using EM not converging

I have found a lot of questions related to EM but nothing specific to my question. I am using the EM algorithm to fit the pLSA model. As far as I can tell (multiple rounds of checking the code) I can ...
2
votes
1answer
47 views

Reason for using log-likelihood in EM algorithm

When I learned EM algorithm, I saw many literatures use (the expectation of ) the log-likelihood. Is there any reason other than that the log-likelihood may reduce computation? Thanks!
1
vote
0answers
10 views

Cross Validation: Which classifier to use in the end - more difficult setting with the EM algorithm

Referring to already discussed question, I solve something more difficult. During the cross validation, I obtain say $n$ models. The discussed question assumes that the best way is to train a new ...
2
votes
0answers
56 views

Likelihood maximization: MCEM algorithm versus MCMC algorithm

Hello Everyone this is my first question. I am a particle physicist and I am doing some empirical studiues on parameters estimation using different methods (this might give me some handle to study on ...
1
vote
0answers
26 views

why is the incomplete log-likelihood difficult to optimize

I am trying to teach myself the expectation-maximization algorithm and the texts say the EM is particularly useful when the incomplete log-likelihood i.e. $P(X|\theta)$ where $\theta$ are the ...
5
votes
1answer
101 views

Question on how to use EM to estimate parameters of this model

I am trying to understand EM and trying to infer parameters of this model using this technique but am having trouble understanding how to begin: So, I have a weighted linear regression model as ...
3
votes
1answer
48 views

Deriving K-means algorithm as a limit of Expectation Maximization for Gaussian Mixtures

Christopher Bishop defines the expected value of the complete-data log likelihood function (i.e. assuming that we are given both the observable data X as well as the latent data Z) as follows: $$ ...
0
votes
1answer
45 views

Update Rules in Expectation Maximization

I am emulating a certain PDF behaviour using a function. However, due to divergent improper integral, I don't have a closed form expression for the normalization constant. To get the PDF, I just ...
0
votes
1answer
139 views

“The EM algorithm failed to converge in 25 iterations”

When I Replace Missing Values - Expectation-Maximization in SPSS, I receive the following message: The EM algorithm failed to converge in 25 iterations. Should the algorithm be able to converge? Can ...
0
votes
0answers
30 views

Contribution to the components of a Gaussian mixture by data features

My question is about modelling data with a GMM using EM. One can split the mean and variance of each component into parts as well when working with data with multiple features. My question is what ...
3
votes
2answers
111 views

Self-study: Finding the maximum likelihood estimates of the parameters of a density function

Consider a random sample $x_1,x_2,...,x_n$ from a newly-generated distribution, whose probability density function is given below \begin{equation} f(x|\alpha,\beta,\sigma)=\frac{1}{\Gamma \left( ...
0
votes
0answers
9 views

Clusteriod questions

I would like to clear some things up because I'm confusing everything. A $clusteriod$ is a coordinate for the mean value of a cluster? So if I have a 2-d .csv file I wish to perform kmeans, the ...
1
vote
0answers
57 views

Help with an exponential function with an indicator and using the EM Algorithm

Two bulbs, Brand A and Brand B, in which their lifetimes are distributed exponentially with expectations $\lambda$ and $\mu$ respectively. They pair $X_i$ and $Y_i$. In the ith experiment, instead ...
0
votes
1answer
48 views

Questions revolving GMM & EM

I am currently reading about the guassian mixture model and the expectation–maximization algorithm. From what I am reading the two differences between the two here is what I've come up with so far, ...
0
votes
0answers
33 views

Convergence Time of the EM Algorithm Depending on the Inital Parameter Values

I try to get an intuitive understanding of the convergence properties of the EM-Algorithm. I wrote a code that does the following experiment. We are given three coins: $H$, $A$ and $B$; with ...
1
vote
1answer
31 views

Unnatural clustering with known clusters shapes and optimization criteria

My question is similar to this question Clustering with shape prior, but with additional information. The second answer suggests a mixture model approach to this problem, which is something like ...
0
votes
0answers
8 views

EM convergence when using em.hmm from PLIS

I use em.hmm function from PLIS package. I tried it on dimensions in range from 2 to 6. In every case of provided data (z-values) EM algorithm does not converge for dimensions 2, 5, 6. So, I wonder ...
0
votes
0answers
58 views

Expectation Maximisation Algorithm: Understand through numeric example

I am trying to learn machine learning concepts through online materials. I just studied tutorial on Expectation Maximisation algorithm. I thought one numerical example can make better understanding. ...
0
votes
1answer
27 views

What is the values of the $P(a)$ and $P(b)$ here?

I am watching a video on EM algorithm here. It gives an example of how EM algorithm works. At first two Gaussian distributions are randomly given, and then by iterative calculations their parameters ...
1
vote
1answer
37 views

Does EM algorithm increase the lower bound as well as true likelihood

I am using a variational bayes method (without a M step since no parameters) to infer my model. My question is, if it is working correctly will it increase the log likelihood of the data, ...
0
votes
0answers
23 views

Expectation maximisation for right-censored iid data from Normal

This is the data (which are length of ropes), $\textrm{Data}=\{99, 70, o ,89, 88, o, 88,70, o ,o\}$, where $o$ are censored data with value above $100$. Assume that data are from $\textrm{iid} \sim ...
0
votes
0answers
17 views

r package for ecpectation maximization with probabilitis for each cluster

In r package i'm using EM algorythm.once it is completed i get latent variable z that assigns each observation a distinct cluster. i'd like to know what are the probabilities for each cluster in a ...
0
votes
2answers
220 views

Single EM imputation with R (using Amelia or other packages)

I am trying to impute missing values with R. I would like to use the EM algorithm for that. As it seems this algorithm is implemented in the ...
2
votes
0answers
95 views

Expectation-Maximization with dependent latent variables

Deriving the equations for a Expectation Maximization over my model, I end up with a posterior for the latent variables (E-step) that prevents me from going on. Generative model I assume my data is ...
1
vote
0answers
43 views

Baum Welch and a 1 state Markov model?

I'm using the Baum-Welch algorithm to determine the parameters of a 2 state Hidden Markov Model. It determines fairly well. When I increase the sample size, the estimations get more concentrated, and ...
1
vote
0answers
16 views

best theory on fitting mixture of gaussians

What are the current best results on fitting mixtures of Gaussians with any algorithm (EM or something fancier)? Specifically, if I know only the number of components, what are the sharpest sample ...
0
votes
0answers
29 views

Number of components in EM

How to find number of components that I need to use in expectation-maximization? The only thing that I can think of is to do a cross validation for each number of components. Is there a better way? ...
0
votes
0answers
19 views

plsa using maximum a posteriori

I have performed topic modeling by PLSA using maximum likelihood estimation. Now I need to perform using maximum a posteriori by using some prior distribution. The prior distribution consists of word ...
3
votes
1answer
173 views

K-means as a limit case of EM algorithm for Gaussian mixtures with covariances $\epsilon^2 I$ going to $0$

My goal is to see that K-means algorithm is in fact Expectation-Maximization algorithm for Gaussian mixtures in which all components have covariance $\sigma^2 I$ in the limit as $\lim_{\sigma \to 0}$. ...
0
votes
0answers
162 views

Choosing initial transition and emission probabilities while training HMM

A Hidden Markov Model (HMM) is defined by the following parameters: $HMM =(prior, transmat, obsmat)$ Using K Murphy's HMM toolbox [1], I ran a small experiment where I define a set of true ...
3
votes
0answers
62 views

Is this problem Bayesian? And can I use variational approximation?

Suppose there are $N$ samples of observations $\mathbf X(n)$ ($n=1,\cdots,N$), which are given by probability distribution $p(\mathbf X(n)|\mathbf Z(n))$ with their conditions are given by hidden ...
0
votes
0answers
82 views

Record Linkage Using Fellegi-Sunter Model

I am trying to create a record linkage system using the fellegi-sunter model.I am following this paper http://digital.library.okstate.edu/etd/SHIN_okstate_0664M_10668.pdf. I am not understanding ...