An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.

learn more… | top users | synonyms (1)

0
votes
0answers
12 views

Looking for real example of Expectation Maximization algorithm to cluster text documents

All I can find is extremely complex mathematical equations and equations and equations, but not single example that explains how the algorithm works expect coins. I can't relate coins and thousands of ...
0
votes
0answers
12 views

Gaussian Mixture models

Can someone explain pdf of mixture models,I do not understand this completlly.I they say that component density is normal,so it means it has normal distribution,yes? Probability density function is ...
2
votes
0answers
24 views

Why isn't a gaussian mixture prone to overfitting?

Consider a Gaussian mixture of 2 components and a dataset of size $N$. The EM algorithm use the data to estimate: the model parameters: the means $\mu_1, \mu_2$ (say the covariances matrices are ...
0
votes
0answers
22 views

Does the expectation maximization algorithm apply to this problem?

I have a sample of variables $x_i$, where each one is a function of known variables $y_i$ and $b_i$, and of an unknown variable $\alpha_i$. $$x_i=y_ib_i-y_i^{\alpha_i}$$ The density function of ...
0
votes
0answers
10 views

Truncated/censored AR1 normal likelihood

I have a model for some data that I am analysing which is of the form: $W^*_t = \rho W^*_{t-1} + \epsilon_t$ Where $\epsilon_t\sim N(0,\sigma^2)$. $W^*_t$ is a latent (hidden) process, which ...
0
votes
0answers
53 views

Confusion with EM Algorithm for Gaussian Mixture?

I am trying to learn EM Algorithm for Gaussian Mixture. But not able to understand few stuffs. This is what I have understood. Consider GMM with k components. $$ p( \mathbf{x}| ...
1
vote
0answers
12 views

EM algorithm for Blind separation and deconvolution of noisy signals using mixture models

I have a question from the paper: Maximum likelihood for blind separation and deconvolution of noisy signals using mixture models (pdf). In section 2, equations 7, 8, 9 read: \begin{align} ...
0
votes
0answers
9 views

Techniques to improve GMM-based binary classifier performance?

I'm currently training a set of 2 GMMs (Gaussian Mixture Models) for 2 classes of data (stressed speech vs neutral speech), and making classification decisions by comparing the posterior probability ...
3
votes
1answer
53 views

Shrinkage of the Sample Covariance matrix

Assume we have N independent and identically distributed random vectors $X_1, X_2, ..., X_N$ where each of them is of size p $\times$ 1. The sample covariance matrix, denoted here by $S$, is computed ...
1
vote
1answer
16 views

What should be the covariance matrices and weights for initializing EM/GMM with kmeans?

It's typical to initialize EM for Gaussian Mixture Models using the result of kmeans clustering. However, kmeans only gives you the means (centers) of the starting GMM, but EM initialization often ...
0
votes
1answer
34 views

How do I “split” Gaussian mixture components when training EM/GMM based classifier?

In order to improve performance of my Gaussian Mixture Model based classifier, I was recommended to start with a single multivariate Gaussian, estimate its parameters, and "split" it into two ...
1
vote
0answers
52 views

Expectation Maximization clarification questions

I know there are many questions related to EM, but I still could not find the answer I seek. I have learned Expectation Maximization from this turtorial: Andrew Ng. The lecture is easy to follow. ...
1
vote
3answers
64 views

Predicting score in the presence of latent variables

Given a dataset with the attributes (hour_of_day, day_of_week, performance) where performance is a function of ...
0
votes
0answers
15 views

why use weighted sum of component densities in unsupervised parametric estimation?

In unsupervised parametric estimation why do we take $p(x|\Theta)$ as $ \sum_j P(w_j)p(x|w_j ,\Theta_j ), j = 1 - c$? That is weighted sum of component densities. where $p(x|\Theta)$ is mixture ...
0
votes
1answer
18 views

Rationale for MCEM

From what I've read, the main advantage of the EM algorithm is that the expectation step can be expressed in closed form giving a deterministic answer and thus 0 variance. What's the rationale then ...
0
votes
0answers
14 views

convergence assumptions for expectation maximization

in andrew ng lectures notes for expectation maximization, i believe the only assumption invoked for the convergence of the EM algorithm is the jensen inequality that operates on the function Log(x), ...
1
vote
1answer
58 views

EM algorithm special case

I'm considering a collection of Bayes Factors, $\mbox{BF}(j)$, $j=1, ..., J$, so that the overall evidence against $H_0$ is represented by the overall Bayes Factor $$\frac{P(x|H_0 \mbox{ false})}{P(x ...
1
vote
0answers
19 views

Criterion to determine the number of kernels in Expectation-Maximization algorithm

Suppose we have a random variable $X$, whose distribution is a Gaussian mixture $$f_K =\sum_{i = 1}^K \alpha_i N(\mu_i, \sigma^2_i)$$ where $\alpha_i> 0$, $\sum_i \alpha_i = 1$ and ...
0
votes
0answers
29 views

How to design an objective function in Convolutional neural networks to classify unlabeled images?

Convolutional neural networks have been used in supervised learning such that it changes the weights $\Theta$ to minimize $(f(X;\Theta)-Y)^2$. However, for unlabeled data, how does one design a new ...
3
votes
1answer
61 views

Why should one use EM vs. say, Gradient Descent with MLE?

Mathematically, it's often seen that expressions and algorithms for Expectation Maximization (EM) are often simpler for mixed models, yet it seems that almost everything (if not everything) that can ...
0
votes
1answer
45 views

Real time example: Estimation for incomplete data

Following is from Csiszar and Shields' FnT monograph "Information Theory and Statistics": The expectation–maximization or EM algorithm is an iterative method frequently used in statistics to ...
0
votes
0answers
26 views

How to reestimate GMMs in a HMM-GMM

Context: Automatic Speech Recognition I understand the training of a pure HMM with Baum-Welch: Expectation step compute $\gamma_t(i) = P(q_t=i |O,\lambda)$ //p(passing state $i$ at frame ...
2
votes
1answer
86 views

MCEM algorithm in normal distribution

Consider $z_1,\ldots,z_n$ as a sample of observations of $Z$ and $y_1,\ldots,y_n$ the missing data, where $Z\sim N(\mu,\sigma^2+1)$ and $Y\sim N(0,1)$. i)Find the expression of ...
1
vote
2answers
48 views

EM algorithm to impute missing value for one variable

This is from Robert Hogg's Introduction to Mathematical Statistics 6th, exercise 6.6.5. p366, It says, Suppose $X_1$, $X_2$, $X_{n1}$, are a random sample from a $N(\theta,1)$ distribution. Suppose ...
0
votes
0answers
20 views

Calculate parameter standard errors when using E-M algorithm?

When using the E-M algorithm, how can we get standard errors for estimated parameters? If we were just maximizing log-likelihood, then hessian=-observed information, then we can get variance from ...
3
votes
1answer
57 views

I am having trouble understanding EM gradient algorithm can someone help?

I understand the formula for the EM gradient algorithm but I don't understand what it means by taking one step of Newton's method. Does it mean that it just take one iteration to converge to the ...
2
votes
1answer
88 views

Convergence of EM for Mixture of Gaussians

Is the Mixture of Gaussians model (an example of latent class analysis) gauranteed to converge on a viable solution even on Unimodal data using the Expectation Maximization algorithm to estimate the ...
2
votes
0answers
57 views

Numerical Approxmation of standard errors for parameter estimation in the EM algorithm

Generally, when you want to compute standard errors for estimated parameters within the ML framework, one uses the diagonal elements of the observed information matrix. In for instance ...
1
vote
1answer
22 views

How do you differentiate a function in the em-algorithm?

I am conducting the EM algorithm. I understand the algorithm and my question is more related to the differentiation procedure within the algorithm more than the algorithm itself. Through using the ...
2
votes
0answers
31 views

Is there any alternative to the EM algorithm?

I am working on biomedical signal analysis and the most used method for parameters estimation is the EM algorithm. My question is : what are the most powerful alternatives to this algorithm?
0
votes
1answer
34 views

Regularizing soft kmeans with entropy

So in classical fuzzy k-means clustering, the objective function is $\sum_i \sum_j u_{ij} \|x_i - c_j\|^2$ Now, we want to regularize this objective function using the entropy: $\sum_i^n H(U_i) = - ...
1
vote
0answers
27 views

log multivariate normal differentiation with VAR process

I am trying to estimate a regime switching model with an autoregressive component using the EM algorithm. The process itself can be presented this way: $$ r_{t}= A_{n \times (n+1)} \boldsymbol ...
2
votes
1answer
48 views

Prove the loglikelihood is strictly concave for ABO allele frequency blood type data

I am working through the problems in Kenn Lange's book Numerical Analysis for Statisticians. I am going to try and do all of the problems in the book, though none of them are specifically assigned for ...
0
votes
0answers
25 views

Reference request: EM algorithm and hidden Markov model books with solutions

I am studying missing data problems and the applications of the EM algorithm to missing data problems, like mixture models and hidden Markov models. We have been using Schafer's book Analysis of ...
0
votes
0answers
38 views

Quasi-Newton Accelerator (QN1) for EM Algorithm

I am trying to implement what is called a "very simple to implement" accelerator for the EM algorithm. Specifically I am talking about the QN1 algorithm, described here, and am using a multivariate ...
0
votes
0answers
20 views

Self-study (Expectation Maximization on Bivariate Normal Distribution)

I see this example is also "classic", and I am attempting to understand how to approach it. I have an iid sample drawn from a bivariate normal distribution with mean vector ($\mu_1, \mu_2$) and ...
0
votes
0answers
13 views

How to fix co variance matrix going singular in Gaussian mixture model implementation?

I am implementing GMM in Matlab without using any machine learning library. I am able to initialize the parameters, perform expectation and maximization for one iteration; but when I put expectation ...
1
vote
1answer
109 views

Expectation-Maximization Algorithm for Binomial

I have a multinomial distribution with four outcomes, with a pdf: $$p(x_1,x_2,x_3,x_4)=\frac{n!}{x_1!x_2!x_3!x_4!}p_1^{x_1}p_2^{x_2}p_3^{x_3}p_4^{x_4}, \sum_{i=1}^4x_i=n, \sum_{i=1}^4p_i=1$$ The ...
0
votes
0answers
21 views

Fast soft-assign to a large high-dimensional GMM

We wish to perform a fast (possibly approximate) soft-assign of high-dimensional data points to a large Gaussian mixture with diagonal covariance (this is related to the E step of the EM algorithm). ...
2
votes
0answers
55 views

EM algorithm: With prior vs. not prior

I have a working EM algorithm without prior. I am asking for some advice on how to add prior on latent variables. Define: $t_i \in \{ +1, -1 \} $: variables of interest to be predicted $p_j \in ...
5
votes
1answer
123 views

What will be the estimator for these parameters

Question: $y_0 = z^d$ is computed from the sum of some recordings by a sensor. Let, there be $k$ sensor nodes. This parameter is calculated by each sensor node and then transmitted to the base ...
1
vote
3answers
40 views

When does the EM for Gaussian mixture model has one of the Gaussian diminish to exactly one point and have zero variance?

I had implemented the EM algorithm for mixture models as follows: For the E-step I compute the soft-counts of assigning each point $x^{(t)} \in Data_n$ to an individual cluster $j \in \{1, ..., K \}$ ...
1
vote
0answers
42 views

How do you do EM algorithm for a factored model for a recommender system?

Let $X$ be a $n \times d$ matrix with users as rows and movies as columns. Each user is a single row $x^{(u)} \in \mathbb{R}^d$ (i.e. for user u there are at most d ratings for the d movies). Also ...
5
votes
1answer
205 views

Convergence from the EM Algorithm with bivariate mixture distribution

I have a mixture model which I want to find the maximum likelihood estimator of given a set of data $x$ and a set of partially observed data $z$. I have implemented both the E-step (calculating the ...
2
votes
1answer
52 views

Expectation Maximisation

I'm currently reading Thomas Hofmamms paper on Probabilistic Latent Semantic Analysis. He includes a formula for the E step in Expectation Maximisation, but has proposed an alternative to this step ...
3
votes
2answers
177 views

Kalman Filter Expectation Maximization

I'm not very familiar with the EM algorithm for the Kalman Filter. I've been using pykalman to do my analysis in Python. The package comes with a simple EM algo: ...
0
votes
1answer
24 views

Convergence of EM in Mixture Models w.r.t unlikely events $(f(\cdot)=0)$ in either distribution

To maximize the likelihood of a mixture model with unobserved latent variables, the Expectation Maximization is conventionally applied. Assuming we have data $x_1,\dots,x_n$ from a fixed number of ...
1
vote
0answers
30 views

Non-Causal time-series filtering techniques for standard noise with unkown variance. (EM vs. weiner vs. kalman)

This is a quick question about filtering stored time-series data using kalman/weiner filtering techniques or expectation maximization. I'm just hoping to fix some confusion about questioning ...
0
votes
0answers
32 views

How to evaluate the goodness of Fit of parameters obtained from EM algorithm

I have a set of observations $\mathcal{Y} = {Y_1, \cdots, Y_T}$. I am running EM algorithm to fit the observations to the following Hidden Markov Model $$A = [a_{ij}]_{N \times N}, a_{ij} = P(X_{k+1} ...
2
votes
0answers
44 views

Beginner level: How to plug in the smoothing equations into E step (Part 2)

Considering Gaussian Linear Dynamical system, $x_{t+1} = Ax_t + w_t$ $y_t = Cx_t + v_t$ $w_t = N(0,Q)$, $v_t = N(0,R)$ By Kalman Filter we are estimating the state variables and the state estimate ...