An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.

learn more… | top users | synonyms (2)

0
votes
2answers
22 views

Expectation Maximization-Log Likehood interpretation

I am using EM algorithm in weka for genomic data, get the results in the images, but a don't know how interpret the log likehood index. Is better when is higher or lower, negative or positive. How ...
1
vote
0answers
26 views

EM versus other methods of optimization

What are some good examples of likelihoods which are easily maximized by EM but not by other methods of optimization (e.g., gradient ascent) and vice versa?
3
votes
0answers
27 views

Indirect solution for maximum entropy through sampling?

Is there a way to sample from a finite set $\{A,B,C,D\}$ such that the limiting empirical proportions converges to the maximum entropy solution of their probabilities consistent with known ...
0
votes
0answers
19 views

EM algorithm help - Plot of expectation

Can someone plot an expectation of a function and show me how maximizing it = maximizing the lower bound of its likelihood in the EM algorithm ? I don't know how to plot the expectation of a function ...
0
votes
0answers
17 views

EM Algorithm - E step

In the E-step, why are there 2 different thetas ? Isnt the expectation of a function $ E[x] = \int xp(x) ?$ If I know x then I would know p(x) as well right ? Based on what I am reading $ E[x] = \int ...
0
votes
0answers
22 views

Question regarding expectation of log in EM

I am reading about the EM algorithm and I have 2 questions to ask. 1) The 1st line states that $ \sum p(x,z|\theta)] = $ the expectation of $ p(x|z)$. Is it because $E[f(x)] = xf(x) ?$ And because ...
1
vote
0answers
15 views

How to make sense of the EM algorithm expressed in terms of Kullback-Leibler divergence?

In the textbook by Wasserman, All of Statistics, the Algorithm is expressed as: 1) Pick a starting value $\theta^0$. 2) (E-Step). Calculuate: $$ J(\theta|\theta^j) = E_{\theta^j} \left(log ...
2
votes
0answers
12 views

EM algorithm increase after E step?

It might be a silly question, but here it goes. The short version of my question is whether the marginal likelihood calculated after every E steps should be increasing or not. More details: Using ...
0
votes
0answers
27 views

understanding E step of EM for GMM

I'm reading this chapter about EM (9.3.1) of the book "Pattern Recognition and Machine Learning". I understand the basic EM algorithm for GMM, but I'm having some problems understanding the ...
0
votes
0answers
32 views

Expectation Maximization (EM) for 3-dimensional parameter $(\alpha,\mu_2,{\sigma}^2)$

Let $x_i$ where $i=1,...,100$ are iid observations from a mix of two normal distributions with means $\mu_1=0$ and $\mu_2$ and the same variance ${\sigma}^2$. If $\alpha$ is the proportion of the ...
0
votes
0answers
15 views

Strategy for estimating a more complex hidden markov model (HMM)

I have a HMM in mind where emission probabilities change over time (not dependent on state). For example, suppose I have two states and four possible emissions. If in state 1, emission probabilities ...
0
votes
1answer
58 views

Compute smoothed probabilities for EM algorithm [closed]

In order to compute the expected value of log-likelihood in EM algorithm, we use 3 different probabilities Forecast (predictive) probabilities Inference probabilities Smoothed probabilities ...
1
vote
1answer
45 views

ZIP Fit Indices Calculated from an EM Algorithm

I am working through @ben-bolker's owls example available here:https://groups.nceas.ucsb.edu/non-linear-modeling/projects/owls/WRITEUP/owls.pdf In particular, I am making use of the R ...
0
votes
0answers
15 views

Is incremental EM a special case of SAGE?

I have come across "A View of the EM Algorithm that Justifies Incremental, Sparse, and Other Variants" (Neal & Hinton, 1998). I was wondering if incremental update in EM as described here can be ...
0
votes
0answers
10 views

How to ensure sample independence across iterations when using the EM algorithm?

I have a problem which uses the EM algorithm. For my purpose, I would like to run it such that the samples used in the $t$ iteration are independent from those used in the $t+1$ iteration. One way to ...
1
vote
0answers
41 views

observed data log likelihood and complete data log likelihood estimation

I have the following model. $p(z_{n}) = Categorical(\pi)$ $p(\pi) = Dir(\alpha)$ $p(x_{n}| z_{n}=k,\mu) = \prod_{d=1}^{D} {Bernouli(\mu_{kd})} $ $p(\mu_{kd}) = uniform(0,1)$ How can I find ...
0
votes
0answers
31 views

Fleshing out specific equation in “Introducing Monte Carlo Methods with R”

Page 157 of "Introducing Monte Carlo Methods with R" reads: A specific solution (Geyer and Thompson, 1992) consists in using only the conditional distribution $k(\pmb{ z ...
0
votes
1answer
47 views

Log likelihood in EM Algorithm

I try understand the log likelihood in weka. I read about that is a probabilistic metric, but i cant understand, if is better when have low value or high value? How i can get the likelihood value, ...
1
vote
0answers
32 views

Maximum likelihood of coin toss of different type?

I was self-studying EM (Expectation Maximization) algorithm, where I came across this example given by the paper. In this paper, there are two types of coins A, B with unknown parameters $θ_A$ and ...
1
vote
1answer
53 views

Derivative in EM Algorithm?

The pic is from Andrew Ng's Machine Learning Class. It's about derivation in EM algorithm. I am not sure how to transfer the second line to the third line. Anyone has some good idea?
0
votes
0answers
25 views

Derivation of expectation maximization algorithm

I am reading this (http://ai.stanford.edu/~chuongdo/papers/em_tutorial.pdf) tutorial on expectation maximization algorithm, I am not able to understand second step. I have already read this ...
1
vote
0answers
74 views

How can I create a topic model with a mixture of multinomials and EM?

I'm trying to create a topic model with a mixture of multinomials and the EM algorithm. I do not want to use a package. For reference, I'm implementing this in Python with numpy. Data Sets I have ...
0
votes
0answers
24 views

Expected sufficient statistics in expectation maximization

In the expectation step of the EM algorithm we impute the expected values of the sufficient statistics of the latent variables. Why do we impute these rather than the expected values of the latent ...
6
votes
1answer
193 views

Help in Expectation Maximization from paper :how to include prior distribution?

The Question is basd on the paper titled : Image reconstruction in diffuse optical tomography using the coupled radiative transport–diffusion model Download link The Authors apply EM algorithm with ...
0
votes
0answers
24 views

Approximating the conditional expectation in simulations

I am simulating stock returns, which are governed by the following equations $r_t = \mu + \delta r_{t-1} + \varepsilon_t$ $\sigma^2_t = \omega + \alpha \varepsilon_{t-1}^2 + \beta \sigma^2_{t-1}$ ...
1
vote
1answer
51 views

I want to show a local optimum in my paper, how do I generate the data for it?

I'm writing a paper where I am explaining the problems of local optimum in my clustering algorithm. While clustering, in my data I would at times get local optimums. But I've tried and I cannot ...
1
vote
1answer
43 views

Clarification on EM Algorithm

So the general set-up for the EM algorithm is the following recursion \begin{align} ...
0
votes
1answer
44 views

EM algorithm with dependent observations

I am trying to implement an EM algorithm for dependent observations. Specifically, I am dealing with families where the hidden variables $Z$ of the children are dependent on the hidden variables of ...
2
votes
0answers
36 views

What is a better political voting method? [closed]

We are in a season where some major elections are happening (e.g. U.S. elections) and I find it interesting to address. Objective When we decide "better", we need to define an objective. To be ...
1
vote
1answer
66 views

K-means algorithm's EM “Maximization” step

I'm a software engineer and am trying to understand how Lloyd's K-Means algorithm fits into the general framework of the Expectation-Maximization (EM) algorithm. I previously read the question ...
0
votes
1answer
28 views

Multinomial logistic regression where two choices are pooled/censored in the data?

I am looking for a lead on estimating a specific type of multinomial choice model. Specifically, assume that I see $N$ people and those people have some vector of characteristic $X$. For ten ...
2
votes
2answers
278 views

Initializing EM algorithm [duplicate]

I am using EM algorithm to estimate measured data ($y$) as a sum of two weighted gaussian distributions: $$model = \sum \limits_{i=1}^{L=2} w_i \phi(\theta_i)$$ Where $\theta$ = ($\mu$, $\sigma$). ...
0
votes
1answer
104 views

Interpreting Cluster Analysis from SAS Enterprise Miner

I am currently doing a text mining project and I conducted a clustering analysis in SAS enterprise miner. I am using the following settings: Anyway, The results look like this, showing me ...
2
votes
3answers
107 views

Is it possible to see the slow decreasing in test negative-log-likelihood as overfitting?

We have developed a model for some real data and we use EM algorithm for optimization of the model (parameters). In first phase we generate synthetic data according to the model (with some known ...
0
votes
0answers
42 views

Does the EM algorithm converge exactly where a grid search on the marginalized likelihood converges?

I have successfully implemented a grid search algorithm to estimate two parameters of a likelihood. I computed the likelihood $l(X;\theta)$ of observed data $X$ by integrating out the discrete ...
0
votes
0answers
15 views

Iterative Maximization issue in Truncated Negative Binomial Regression in Stata

I was running truncated negative binomial regression in Stata and got a problem. During the iteration process, my results show " backed up" at the end of final iteration which means Stata could not ...
3
votes
0answers
53 views

Cross validation or EM for selecting strength of the prior?

Often when I'm looking at bayesian analyses, the influence of the prior is chosen via cross validation. For example, suppose $X$ and $Y$ represent some real valued data that I want perform a bayesian ...
1
vote
1answer
37 views

When would I use EM instead of k-means?

When would I want to assign cluster probabilities to patterns instead of hard assignments to clusters? Can someone elaborate?
4
votes
0answers
47 views

How to define a likelihood function for an EM algorithm

Assuming $A$ a set of vectors from a normal distribution, and $X$ a projection matrix and $B$ a set of projected vectors of $A$ using $X$: $B=A*X$ Using an EM approach and by initializing X from ...
3
votes
1answer
45 views

E-step in EM algorithm with non trivial latent variables

I am trying to derive the E-step for an EM algorithm for this model: The interesting fact is that there are two sets of latent variables: $z$ and $y$. The E-step involve a derivation of the ...
1
vote
0answers
17 views

What are the generating distributions in EM?

I have $N$ latent variables $Z$ and $K$ latent variables $X$. $Z_n \sim \text{Cat}(\gamma^n_{1:6})$ $X_k \sim \text{Cat}(\theta^k_{1:6})$ These have categorical distributions for the outcomes ...
0
votes
0answers
33 views

EM bound for factor analysis, imaginary?

I am implementing factor analysis using EM. I get the likelihood, always increasing as expected. However, when I try to get the lower bound for which we have a closed form that allows us to keep ...
3
votes
1answer
198 views

ML covariance estimation from Expectation-Maximization with missing data

Assuming a multivariate normal distribution with missing data, is there a straightforward way to find the maximum likelihood estimate for covariance using an Expectation-Maximization algorithm? NOTE: ...
0
votes
0answers
36 views

Expectation Maximization with coin flips

I'm trying to understand how the description of EM provided here is related to the more general description (wiki). The experiment is as follows: there are two biased coins A and B. We do the ...
2
votes
1answer
38 views

How does the EM algorithm operate when group label may be missing?

I have a set of data where the group label for a bunch of the data is missing. I know that there are 10 groups (integer 1 to 10): ...
1
vote
0answers
26 views

E and M step for applying EM on zero inflated Poisson Regression [duplicate]

$$L(\gamma,\mathbf{\beta}; \mathbf{\gamma}, \textbf{z}) = \sum_{i=1}^{n} \log(f(z_i|\mathbf{\gamma}))+\sum_{i=1}^{n} \log(f(y_i|z_i, \mathbf{\beta}))$$ $$= \sum_{i=1}^{n} (z_{i} \textbf{G}_i ...
1
vote
1answer
31 views

Expectations versus returns

I have calculated the probability of winning an event, say a horse race, and am comparing it with the odds on offer. I have two horses which look good prices. My model says horse A has a 5% chance ...
0
votes
0answers
23 views

Maximum Likelihood Estimator using EM Algorithm with missing data

Suppose you have in total 1000 observations of height and weight of people from 10 different countries. If the country of a person is known, then the height and weight of him are independently normal ...
0
votes
0answers
21 views

Calculating 'k' for k-Means and Expectation Maximization

This question inspired my question. I've read a lot of articles on the Internet, and it seems like most people use sums of squares to find 'k' for k-Means and they use BIC to find 'k' for Expectation ...
1
vote
1answer
66 views

EM and Genetic Algorithms

Apologize in advance for intentional vagueness. Questions I'd like answered highlighted in bold. Intro So we have an algorithm which, given a weighting function and an item to process, processes the ...