An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.

learn more… | top users | synonyms (2)

0
votes
0answers
9 views

Iterative Maximization issue in Truncated Negative Binomial Regression in Stata

I was running truncated negative binomial regression in Stata and got a problem. During the iteration process, my results show " backed up" at the end of final iteration which means Stata could not ...
3
votes
0answers
44 views

Cross validation or EM for selecting strength of the prior?

Often when I'm looking at bayesian analyses, the influence of the prior is chosen via cross validation. For example, suppose $X$ and $Y$ represent some real valued data that I want perform a bayesian ...
1
vote
1answer
27 views

When would I use EM instead of k-means?

When would I want to assign cluster probabilities to patterns instead of hard assignments to clusters? Can someone elaborate?
4
votes
0answers
39 views

How to define a likelihood function for an EM algorithm

Assuming $A$ a set of vectors from a normal distribution, and $X$ a projection matrix and $B$ a set of projected vectors of $A$ using $X$: $B=A*X$ Using an EM approach and by initializing X from ...
3
votes
1answer
39 views

E-step in EM algorithm with non trivial latent variables

I am trying to derive the E-step for an EM algorithm for this model: The interesting fact is that there are two sets of latent variables: $z$ and $y$. The E-step involve a derivation of the ...
1
vote
0answers
16 views

What are the generating distributions in EM?

I have $N$ latent variables $Z$ and $K$ latent variables $X$. $Z_n \sim \text{Cat}(\gamma^n_{1:6})$ $X_k \sim \text{Cat}(\theta^k_{1:6})$ These have categorical distributions for the outcomes ...
0
votes
0answers
28 views

EM bound for factor analysis, imaginary?

I am implementing factor analysis using EM. I get the likelihood, always increasing as expected. However, when I try to get the lower bound for which we have a closed form that allows us to keep ...
3
votes
1answer
156 views

ML covariance estimation from Expectation-Maximization with missing data

Assuming a multivariate normal distribution with missing data, is there a straightforward way to find the maximum likelihood estimate for covariance using an Expectation-Maximization algorithm? NOTE: ...
0
votes
0answers
33 views

Expectation Maximization with coin flips

I'm trying to understand how the description of EM provided here is related to the more general description (wiki). The experiment is as follows: there are two biased coins A and B. We do the ...
2
votes
1answer
33 views

How does the EM algorithm operate when group label may be missing?

I have a set of data where the group label for a bunch of the data is missing. I know that there are 10 groups (integer 1 to 10): ...
1
vote
0answers
26 views

E and M step for applying EM on zero inflated Poisson Regression [duplicate]

$$L(\gamma,\mathbf{\beta}; \mathbf{\gamma}, \textbf{z}) = \sum_{i=1}^{n} \log(f(z_i|\mathbf{\gamma}))+\sum_{i=1}^{n} \log(f(y_i|z_i, \mathbf{\beta}))$$ $$= \sum_{i=1}^{n} (z_{i} \textbf{G}_i ...
1
vote
1answer
28 views

Expectations versus returns

I have calculated the probability of winning an event, say a horse race, and am comparing it with the odds on offer. I have two horses which look good prices. My model says horse A has a 5% chance ...
0
votes
0answers
20 views

Maximum Likelihood Estimator using EM Algorithm with missing data

Suppose you have in total 1000 observations of height and weight of people from 10 different countries. If the country of a person is known, then the height and weight of him are independently normal ...
0
votes
0answers
20 views

Calculating 'k' for k-Means and Expectation Maximization

This question inspired my question. I've read a lot of articles on the Internet, and it seems like most people use sums of squares to find 'k' for k-Means and they use BIC to find 'k' for Expectation ...
1
vote
1answer
49 views

EM and Genetic Algorithms

Apologize in advance for intentional vagueness. Questions I'd like answered highlighted in bold. Intro So we have an algorithm which, given a weighting function and an item to process, processes the ...
1
vote
1answer
30 views

Details in proof for convergence of Expectation Maximization Algorithm

I am going through the paper provided here http://www.cs.cmu.edu/~dgovinda/pdf/recog/EM_algorithm-1.pdf I could not make out how the following was derived $\sum_z \mathcal P(\mathbf z|X, \theta_n) ...
4
votes
1answer
94 views

Help with Variational Bayes on a weighted linear regression model

I am trying to setup VB to do a weighted linear regression for vector observations. My setup is that I have $N$ numbers of $d$-dimensional vector observations. I would like to model the noise as being ...
2
votes
3answers
82 views

Applying variational inference to this model

I am basically trying to do a weighted linear regression in a bayesian way. This is to ensure that the I can take care of the heretoscedastic noise. So, my model is like: $$ y_i \sim ...
2
votes
1answer
78 views

Variational Bayes: Understanding Mean field approximation

I am looking at the mean field approximation as used in Variational Bayes inference and I looked at this section on wikipedia with the factorised approximation as described here: ...
2
votes
0answers
21 views

Question about M-step for bimodal Poisson Proces

all I have difficulty in deriving the result for the M-step of EM of the bimodal Poisson as shown in paper, Byers, Simon, and Adrian E. Raftery. "Nearest-neighbor clutter removal for estimating ...
1
vote
0answers
20 views

Expectation-Maximization for NLP tasks

I am looking for resources on Expectation-Maximization for NLP tasks. Ideally, they should be mathematically thorough (vs. just take some soft counts here and there), and give some clear intuitions.
0
votes
0answers
33 views

Understanding of a specific EM example

I am reading Computational Statistics by Givens et. al on Chap 4 EM algorithms. In Sec. 4.2 above formula (4.16), it says "Table 4.1 shows how the EM algorithm converges to MLEs". It seems to me that ...
0
votes
0answers
11 views

Expectation maximization from distances

I want to infer location from noisy distance measurements from a sensor network. I've been doing initial simulations and trying to use EM to help. I have: A list of distance measurements to each ...
0
votes
0answers
25 views

Gaussian Mixture models

Can someone explain pdf of mixture models,I do not understand this completlly.I they say that component density is normal,so it means it has normal distribution,yes? Probability density function is ...
2
votes
0answers
49 views

Why isn't a gaussian mixture prone to overfitting?

Consider a Gaussian mixture of 2 components and a dataset of size $N$. The EM algorithm use the data to estimate: the model parameters: the means $\mu_1, \mu_2$ (say the covariances matrices are ...
0
votes
0answers
31 views

Does the expectation maximization algorithm apply to this problem?

I have a sample of variables $x_i$, where each one is a function of known variables $y_i$ and $b_i$, and of an unknown variable $\alpha_i$. $$x_i=y_ib_i-y_i^{\alpha_i}$$ The density function of ...
0
votes
0answers
14 views

Truncated/censored AR1 normal likelihood

I have a model for some data that I am analysing which is of the form: $W^*_t = \rho W^*_{t-1} + \epsilon_t$ Where $\epsilon_t\sim N(0,\sigma^2)$. $W^*_t$ is a latent (hidden) process, which ...
0
votes
0answers
72 views

Confusion with EM Algorithm for Gaussian Mixture?

I am trying to learn EM Algorithm for Gaussian Mixture. But not able to understand few stuffs. This is what I have understood. Consider GMM with k components. $$ p( \mathbf{x}| ...
1
vote
0answers
21 views

EM algorithm for Blind separation and deconvolution of noisy signals using mixture models

I have a question from the paper: Maximum likelihood for blind separation and deconvolution of noisy signals using mixture models (pdf). In section 2, equations 7, 8, 9 read: \begin{align} ...
0
votes
0answers
32 views

Techniques to improve GMM-based binary classifier performance?

I'm currently training a set of 2 GMMs (Gaussian Mixture Models) for 2 classes of data (stressed speech vs neutral speech), and making classification decisions by comparing the posterior probability ...
3
votes
1answer
68 views

Shrinkage of the Sample Covariance matrix

Assume we have N independent and identically distributed random vectors $X_1, X_2, ..., X_N$ where each of them is of size p $\times$ 1. The sample covariance matrix, denoted here by $S$, is computed ...
1
vote
1answer
77 views

What should be the covariance matrices and weights for initializing EM/GMM with kmeans?

It's typical to initialize EM for Gaussian Mixture Models using the result of kmeans clustering. However, kmeans only gives you the means (centers) of the starting GMM, but EM initialization often ...
0
votes
1answer
96 views

How do I “split” Gaussian mixture components when training EM/GMM based classifier?

In order to improve performance of my Gaussian Mixture Model based classifier, I was recommended to start with a single multivariate Gaussian, estimate its parameters, and "split" it into two ...
1
vote
0answers
77 views

Expectation Maximization clarification questions

I know there are many questions related to EM, but I still could not find the answer I seek. I have learned Expectation Maximization from this turtorial: Andrew Ng. The lecture is easy to follow. ...
1
vote
2answers
85 views

Predicting score in the presence of latent variables

Given a dataset with the attributes (hour_of_day, day_of_week, performance) where performance is a function of ...
0
votes
0answers
17 views

why use weighted sum of component densities in unsupervised parametric estimation?

In unsupervised parametric estimation why do we take $p(x|\Theta)$ as $ \sum_j P(w_j)p(x|w_j ,\Theta_j ), j = 1 - c$? That is weighted sum of component densities. where $p(x|\Theta)$ is mixture ...
0
votes
1answer
25 views

Rationale for MCEM

From what I've read, the main advantage of the EM algorithm is that the expectation step can be expressed in closed form giving a deterministic answer and thus 0 variance. What's the rationale then ...
0
votes
0answers
41 views

convergence assumptions for expectation maximization

in andrew ng lectures notes for expectation maximization, i believe the only assumption invoked for the convergence of the EM algorithm is the jensen inequality that operates on the function Log(x), ...
1
vote
1answer
68 views

EM algorithm special case

I'm considering a collection of Bayes Factors, $\mbox{BF}(j)$, $j=1, ..., J$, so that the overall evidence against $H_0$ is represented by the overall Bayes Factor $$\frac{P(x|H_0 \mbox{ false})}{P(x ...
1
vote
0answers
23 views

Criterion to determine the number of kernels in Expectation-Maximization algorithm

Suppose we have a random variable $X$, whose distribution is a Gaussian mixture $$f_K =\sum_{i = 1}^K \alpha_i N(\mu_i, \sigma^2_i)$$ where $\alpha_i> 0$, $\sum_i \alpha_i = 1$ and ...
0
votes
0answers
48 views

How to design an objective function in Convolutional neural networks to classify unlabeled images?

Convolutional neural networks have been used in supervised learning such that it changes the weights $\Theta$ to minimize $(f(X;\Theta)-Y)^2$. However, for unlabeled data, how does one design a new ...
3
votes
1answer
121 views

Why should one use EM vs. say, Gradient Descent with MLE?

Mathematically, it's often seen that expressions and algorithms for Expectation Maximization (EM) are often simpler for mixed models, yet it seems that almost everything (if not everything) that can ...
0
votes
1answer
80 views

Real time example: Estimation for incomplete data

Following is from Csiszar and Shields' FnT monograph "Information Theory and Statistics": The expectation–maximization or EM algorithm is an iterative method frequently used in statistics to ...
2
votes
0answers
86 views

How to reestimate GMMs in a HMM-GMM

Context: Automatic Speech Recognition I understand the training of a pure HMM with Baum-Welch: Expectation step compute $\gamma_t(i) = P(q_t=i |O,\lambda)$ //p(passing state $i$ at frame ...
2
votes
1answer
98 views

MCEM algorithm in normal distribution

Consider $z_1,\ldots,z_n$ as a sample of observations of $Z$ and $y_1,\ldots,y_n$ the missing data, where $Z\sim N(\mu,\sigma^2+1)$ and $Y\sim N(0,1)$. i)Find the expression of ...
1
vote
2answers
106 views

EM algorithm to impute missing value for one variable

This is from Robert Hogg's Introduction to Mathematical Statistics 6th, exercise 6.6.5. p366, It says, Suppose $X_1$, $X_2$, $X_{n1}$, are a random sample from a $N(\theta,1)$ distribution. Suppose ...
0
votes
0answers
41 views

Calculate parameter standard errors when using E-M algorithm?

When using the E-M algorithm, how can we get standard errors for estimated parameters? If we were just maximizing log-likelihood, then hessian=-observed information, then we can get variance from ...
3
votes
1answer
64 views

I am having trouble understanding EM gradient algorithm can someone help?

I understand the formula for the EM gradient algorithm but I don't understand what it means by taking one step of Newton's method. Does it mean that it just take one iteration to converge to the ...
2
votes
1answer
133 views

Convergence of EM for Mixture of Gaussians

Is the Mixture of Gaussians model (an example of latent class analysis) gauranteed to converge on a viable solution even on Unimodal data using the Expectation Maximization algorithm to estimate the ...
2
votes
0answers
106 views

Numerical Approxmation of standard errors for parameter estimation in the EM algorithm

Generally, when you want to compute standard errors for estimated parameters within the ML framework, one uses the diagonal elements of the observed information matrix. In for instance ...