An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.

learn more… | top users | synonyms (2)

0
votes
0answers
8 views

Strategy for estimating a more complex hidden markov model (HMM)

I have a HMM in mind where emission probabilities change over time (not dependent on state). For example, suppose I have two states and four possible emissions. If in state 1, emission probabilities ...
0
votes
1answer
14 views

Expectation maximization for mixture models: where to find implementations? [on hold]

I am looking for a matlab or python implementation of the EM algorithm for learning multivariate mixture models with unspecified distributions, but I can find on line only implementations specific for ...
0
votes
1answer
49 views

Compute smoothed probabilities for EM algorithm [closed]

In order to compute the expected value of log-likelihood in EM algorithm, we use 3 different probabilities Forecast (predictive) probabilities Inference probabilities Smoothed probabilities ...
1
vote
1answer
41 views

ZIP Fit Indices Calculated from an EM Algorithm

I am working through @ben-bolker's owls example available here:https://groups.nceas.ucsb.edu/non-linear-modeling/projects/owls/WRITEUP/owls.pdf In particular, I am making use of the R ...
0
votes
0answers
15 views

Is incremental EM a special case of SAGE?

I have come across "A View of the EM Algorithm that Justifies Incremental, Sparse, and Other Variants" (Neal & Hinton, 1998). I was wondering if incremental update in EM as described here can be ...
0
votes
0answers
9 views

How to ensure sample independence across iterations when using the EM algorithm?

I have a problem which uses the EM algorithm. For my purpose, I would like to run it such that the samples used in the $t$ iteration are independent from those used in the $t+1$ iteration. One way to ...
1
vote
0answers
37 views

observed data log likelihood and complete data log likelihood estimation

I have the following model. $p(z_{n}) = Categorical(\pi)$ $p(\pi) = Dir(\alpha)$ $p(x_{n}| z_{n}=k,\mu) = \prod_{d=1}^{D} {Bernouli(\mu_{kd})} $ $p(\mu_{kd}) = uniform(0,1)$ How can I find ...
0
votes
0answers
31 views

Fleshing out specific equation in “Introducing Monte Carlo Methods with R”

Page 157 of "Introducing Monte Carlo Methods with R" reads: A specific solution (Geyer and Thompson, 1992) consists in using only the conditional distribution $k(\pmb{ z ...
0
votes
1answer
33 views

Log likelihood in EM Algorithm

I try understand the log likelihood in weka. I read about that is a probabilistic metric, but i cant understand, if is better when have low value or high value? How i can get the likelihood value, ...
1
vote
0answers
29 views

Maximum likelihood of coin toss of different type?

I was self-studying EM (Expectation Maximization) algorithm, where I came across this example given by the paper. In this paper, there are two types of coins A, B with unknown parameters $θ_A$ and ...
1
vote
1answer
51 views

Derivative in EM Algorithm?

The pic is from Andrew Ng's Machine Learning Class. It's about derivation in EM algorithm. I am not sure how to transfer the second line to the third line. Anyone has some good idea?
0
votes
0answers
23 views

Derivation of expectation maximization algorithm

I am reading this (http://ai.stanford.edu/~chuongdo/papers/em_tutorial.pdf) tutorial on expectation maximization algorithm, I am not able to understand second step. I have already read this ...
1
vote
0answers
66 views

How can I create a topic model with a mixture of multinomials and EM?

I'm trying to create a topic model with a mixture of multinomials and the EM algorithm. I do not want to use a package. For reference, I'm implementing this in Python with numpy. Data Sets I have ...
0
votes
0answers
23 views

Expected sufficient statistics in expectation maximization

In the expectation step of the EM algorithm we impute the expected values of the sufficient statistics of the latent variables. Why do we impute these rather than the expected values of the latent ...
6
votes
1answer
192 views

Help in Expectation Maximization from paper :how to include prior distribution?

The Question is basd on the paper titled : Image reconstruction in diffuse optical tomography using the coupled radiative transport–diffusion model Download link The Authors apply EM algorithm with ...
0
votes
0answers
24 views

Approximating the conditional expectation in simulations

I am simulating stock returns, which are governed by the following equations $r_t = \mu + \delta r_{t-1} + \varepsilon_t$ $\sigma^2_t = \omega + \alpha \varepsilon_{t-1}^2 + \beta \sigma^2_{t-1}$ ...
1
vote
1answer
49 views

I want to show a local optimum in my paper, how do I generate the data for it?

I'm writing a paper where I am explaining the problems of local optimum in my clustering algorithm. While clustering, in my data I would at times get local optimums. But I've tried and I cannot ...
1
vote
1answer
42 views

Clarification on EM Algorithm

So the general set-up for the EM algorithm is the following recursion \begin{align} ...
0
votes
1answer
44 views

EM algorithm with dependent observations

I am trying to implement an EM algorithm for dependent observations. Specifically, I am dealing with families where the hidden variables $Z$ of the children are dependent on the hidden variables of ...
2
votes
0answers
36 views

What is a better political voting method? [closed]

We are in a season where some major elections are happening (e.g. U.S. elections) and I find it interesting to address. Objective When we decide "better", we need to define an objective. To be ...
1
vote
1answer
56 views

K-means algorithm's EM “Maximization” step

I'm a software engineer and am trying to understand how Lloyd's K-Means algorithm fits into the general framework of the Expectation-Maximization (EM) algorithm. I previously read the question ...
0
votes
1answer
25 views

Multinomial logistic regression where two choices are pooled/censored in the data?

I am looking for a lead on estimating a specific type of multinomial choice model. Specifically, assume that I see $N$ people and those people have some vector of characteristic $X$. For ten ...
2
votes
2answers
276 views

Initializing EM algorithm [duplicate]

I am using EM algorithm to estimate measured data ($y$) as a sum of two weighted gaussian distributions: $$model = \sum \limits_{i=1}^{L=2} w_i \phi(\theta_i)$$ Where $\theta$ = ($\mu$, $\sigma$). ...
0
votes
1answer
65 views

Interpreting Cluster Analysis from SAS Enterprise Miner

I am currently doing a text mining project and I conducted a clustering analysis in SAS enterprise miner. I am using the following settings: Anyway, The results look like this, showing me ...
2
votes
3answers
100 views

Is it possible to see the slow decreasing in test negative-log-likelihood as overfitting?

We have developed a model for some real data and we use EM algorithm for optimization of the model (parameters). In first phase we generate synthetic data according to the model (with some known ...
0
votes
0answers
39 views

Does the EM algorithm converge exactly where a grid search on the marginalized likelihood converges?

I have successfully implemented a grid search algorithm to estimate two parameters of a likelihood. I computed the likelihood $l(X;\theta)$ of observed data $X$ by integrating out the discrete ...
0
votes
0answers
15 views

Iterative Maximization issue in Truncated Negative Binomial Regression in Stata

I was running truncated negative binomial regression in Stata and got a problem. During the iteration process, my results show " backed up" at the end of final iteration which means Stata could not ...
3
votes
0answers
53 views

Cross validation or EM for selecting strength of the prior?

Often when I'm looking at bayesian analyses, the influence of the prior is chosen via cross validation. For example, suppose $X$ and $Y$ represent some real valued data that I want perform a bayesian ...
1
vote
1answer
35 views

When would I use EM instead of k-means?

When would I want to assign cluster probabilities to patterns instead of hard assignments to clusters? Can someone elaborate?
4
votes
0answers
45 views

How to define a likelihood function for an EM algorithm

Assuming $A$ a set of vectors from a normal distribution, and $X$ a projection matrix and $B$ a set of projected vectors of $A$ using $X$: $B=A*X$ Using an EM approach and by initializing X from ...
3
votes
1answer
44 views

E-step in EM algorithm with non trivial latent variables

I am trying to derive the E-step for an EM algorithm for this model: The interesting fact is that there are two sets of latent variables: $z$ and $y$. The E-step involve a derivation of the ...
1
vote
0answers
17 views

What are the generating distributions in EM?

I have $N$ latent variables $Z$ and $K$ latent variables $X$. $Z_n \sim \text{Cat}(\gamma^n_{1:6})$ $X_k \sim \text{Cat}(\theta^k_{1:6})$ These have categorical distributions for the outcomes ...
0
votes
0answers
31 views

EM bound for factor analysis, imaginary?

I am implementing factor analysis using EM. I get the likelihood, always increasing as expected. However, when I try to get the lower bound for which we have a closed form that allows us to keep ...
3
votes
1answer
183 views

ML covariance estimation from Expectation-Maximization with missing data

Assuming a multivariate normal distribution with missing data, is there a straightforward way to find the maximum likelihood estimate for covariance using an Expectation-Maximization algorithm? NOTE: ...
0
votes
0answers
36 views

Expectation Maximization with coin flips

I'm trying to understand how the description of EM provided here is related to the more general description (wiki). The experiment is as follows: there are two biased coins A and B. We do the ...
2
votes
1answer
38 views

How does the EM algorithm operate when group label may be missing?

I have a set of data where the group label for a bunch of the data is missing. I know that there are 10 groups (integer 1 to 10): ...
1
vote
0answers
26 views

E and M step for applying EM on zero inflated Poisson Regression [duplicate]

$$L(\gamma,\mathbf{\beta}; \mathbf{\gamma}, \textbf{z}) = \sum_{i=1}^{n} \log(f(z_i|\mathbf{\gamma}))+\sum_{i=1}^{n} \log(f(y_i|z_i, \mathbf{\beta}))$$ $$= \sum_{i=1}^{n} (z_{i} \textbf{G}_i ...
1
vote
1answer
30 views

Expectations versus returns

I have calculated the probability of winning an event, say a horse race, and am comparing it with the odds on offer. I have two horses which look good prices. My model says horse A has a 5% chance ...
0
votes
0answers
23 views

Maximum Likelihood Estimator using EM Algorithm with missing data

Suppose you have in total 1000 observations of height and weight of people from 10 different countries. If the country of a person is known, then the height and weight of him are independently normal ...
0
votes
0answers
21 views

Calculating 'k' for k-Means and Expectation Maximization

This question inspired my question. I've read a lot of articles on the Internet, and it seems like most people use sums of squares to find 'k' for k-Means and they use BIC to find 'k' for Expectation ...
1
vote
1answer
61 views

EM and Genetic Algorithms

Apologize in advance for intentional vagueness. Questions I'd like answered highlighted in bold. Intro So we have an algorithm which, given a weighting function and an item to process, processes the ...
1
vote
1answer
32 views

Details in proof for convergence of Expectation Maximization Algorithm

I am going through the paper provided here http://www.cs.cmu.edu/~dgovinda/pdf/recog/EM_algorithm-1.pdf I could not make out how the following was derived $\sum_z \mathcal P(\mathbf z|X, \theta_n) ...
4
votes
1answer
98 views

Help with Variational Bayes on a weighted linear regression model

I am trying to setup VB to do a weighted linear regression for vector observations. My setup is that I have $N$ numbers of $d$-dimensional vector observations. I would like to model the noise as being ...
2
votes
3answers
85 views

Applying variational inference to this model

I am basically trying to do a weighted linear regression in a bayesian way. This is to ensure that the I can take care of the heretoscedastic noise. So, my model is like: $$ y_i \sim ...
2
votes
1answer
99 views

Variational Bayes: Understanding Mean field approximation

I am looking at the mean field approximation as used in Variational Bayes inference and I looked at this section on wikipedia with the factorised approximation as described here: ...
2
votes
0answers
25 views

Question about M-step for bimodal Poisson Proces

all I have difficulty in deriving the result for the M-step of EM of the bimodal Poisson as shown in paper, Byers, Simon, and Adrian E. Raftery. "Nearest-neighbor clutter removal for estimating ...
1
vote
1answer
30 views

Expectation-Maximization for NLP tasks

I am looking for resources on Expectation-Maximization for NLP tasks. Ideally, they should be mathematically thorough (vs. just take some soft counts here and there), and give some clear intuitions.
0
votes
0answers
35 views

Understanding of a specific EM example

I am reading Computational Statistics by Givens et. al on Chap 4 EM algorithms. In Sec. 4.2 above formula (4.16), it says "Table 4.1 shows how the EM algorithm converges to MLEs". It seems to me that ...
0
votes
0answers
13 views

Expectation maximization from distances

I want to infer location from noisy distance measurements from a sensor network. I've been doing initial simulations and trying to use EM to help. I have: A list of distance measurements to each ...
0
votes
0answers
32 views

Gaussian Mixture models

Can someone explain pdf of mixture models,I do not understand this completlly.I they say that component density is normal,so it means it has normal distribution,yes? Probability density function is ...