An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.

learn more… | top users | synonyms (1)

0
votes
0answers
6 views

EM convergence when using em.hmm from PLIS

I use em.hmm function from PLIS package. I tried it on dimensions in range from 2 to 6. In every case of provided data (z-values) EM algorithm does not converge for dimensions 2, 5, 6. So, I wonder ...
0
votes
0answers
29 views

Expectation Maximisation Algorithm: Understand through numeric example

I am trying to learn machine learning concepts through online materials. I just studied tutorial on Expectation Maximisation algorithm. I thought one numerical example can make better understanding. ...
0
votes
0answers
33 views
+50

How to formulate the pdf and likelihood function in delay embedded space

PROBLEM STATEMENT: Let $Z$ be random variable in $d$ dimensional space expressed as $\{u_t = (u_t,u_{t+1},...,u_{t+d-1}| t=1,2,..,100\}$; $d=2$. So, $Z$ is obtained from $u$ after Takens Delay ...
0
votes
1answer
25 views

What is the values of the $P(a)$ and $P(b)$ here?

I am watching a video on EM algorithm here. It gives an example of how EM algorithm works. At first two Gaussian distributions are randomly given, and then by iterative calculations their parameters ...
1
vote
1answer
29 views

Does EM algorithm increase the lower bound as well as true likelihood

I am using a variational bayes method (without a M step since no parameters) to infer my model. My question is, if it is working correctly will it increase the log likelihood of the data, ...
0
votes
0answers
14 views

Expectation maximisation for right-censored iid data from Normal

This is the data (which are length of ropes), $\textrm{Data}=\{99, 70, o ,89, 88, o, 88,70, o ,o\}$, where $o$ are censored data with value above $100$. Assume that data are from $\textrm{iid} \sim ...
0
votes
0answers
13 views

r package for ecpectation maximization with probabilitis for each cluster

In r package i'm using EM algorythm.once it is completed i get latent variable z that assigns each observation a distinct cluster. i'd like to know what are the probabilities for each cluster in a ...
0
votes
1answer
47 views

Single EM imputation with R (using Amelia or other packages)

I am trying to impute missing values with R. I would like to use the EM algorithm for that. As it seems this algorithm is implemented in the ...
2
votes
0answers
63 views

Expectation-Maximization with dependent latent variables

Deriving the equations for a Expectation Maximization over my model, I end up with a posterior for the latent variables (E-step) that prevents me from going on. Generative model I assume my data is ...
1
vote
0answers
33 views

Baum Welch and a 1 state Markov model?

I'm using the Baum-Welch algorithm to determine the parameters of a 2 state Hidden Markov Model. It determines fairly well. When I increase the sample size, the estimations get more concentrated, and ...
1
vote
0answers
12 views

best theory on fitting mixture of gaussians

What are the current best results on fitting mixtures of Gaussians with any algorithm (EM or something fancier)? Specifically, if I know only the number of components, what are the sharpest sample ...
0
votes
0answers
20 views

Number of components in EM

How to find number of components that I need to use in expectation-maximization? The only thing that I can think of is to do a cross validation for each number of components. Is there a better way? ...
0
votes
0answers
14 views

plsa using maximum a posteriori

I have performed topic modeling by PLSA using maximum likelihood estimation. Now I need to perform using maximum a posteriori by using some prior distribution. The prior distribution consists of word ...
0
votes
0answers
74 views

K-means as a limit case of EM algorithm for Gaussian mixtures with covariances $\epsilon^2 I$ going to $0$

My goal is to see that K-means algorithm is in fact Expectation-Maximization algorithm for Gaussian mixtures in which all components have covariance $\sigma^2 I$ in the limit as $\lim_{\sigma \to 0}$. ...
0
votes
0answers
61 views

Choosing initial transition and emission probabilities while training HMM

A Hidden Markov Model (HMM) is defined by the following parameters: $HMM =(prior, transmat, obsmat)$ Using K Murphy's HMM toolbox [1], I ran a small experiment where I define a set of true ...
3
votes
0answers
49 views

Is this problem Bayesian? And can I use variational approximation?

Suppose there are $N$ samples of observations $\mathbf X(n)$ ($n=1,\cdots,N$), which are given by probability distribution $p(\mathbf X(n)|\mathbf Z(n))$ with their conditions are given by hidden ...
0
votes
0answers
37 views

Record Linkage Using Fellegi-Sunter Model

I am trying to create a record linkage system using the fellegi-sunter model.I am following this paper http://digital.library.okstate.edu/etd/SHIN_okstate_0664M_10668.pdf. I am not understanding ...
5
votes
1answer
60 views

Hidden Markov Models with multiple emissions per state

I want to use Hidden Markov Models for an unsupervised sequence tagging problem. Due to the peculiarities of my application domain (recognition of dialogue acts in conversations), I would like to use ...
0
votes
1answer
35 views

Mixing probabilities in mixture models using EM

Assume we want to estimate the mixing probabilities ($\pi_{k}$) for each member distribution in the mixture model. We know that $\sum_{m}^{K}\pi_{m}=1$, so we can formulate the optimization problem ...
1
vote
0answers
10 views

unbiased sampling of expecation over maximization operator

My problem setting is as follows, I have a set of M random variable X = \{ {X_1},{X_2},...{X_M}\} where each variable X_i is estimated via stochastic sequence ...
0
votes
0answers
21 views

Expectation Maximization (EM) Method - All Constructs or One Construct at a time?

I want to ask whether I can run Expectation Maximization (EM) method in SPSS to replace ALL missing values of ALL constructs at one run, or I have to do for each construct separately? Thank you
2
votes
1answer
53 views

MAP estimate of posterior parameters

I have a setup where the joint posterior is written as: $$ P(w, \lambda, \phi \vert y) = P(\phi) \times P(w \vert \lambda) \times P(\lambda) \times \prod_{i=1}^{N}P(y_i \vert w_i, \phi, \lambda) $$ ...
1
vote
1answer
68 views

EM for Mixtures of Bernoulli (M-step)

When applying the M-step for a mixture of Bernoulli distributions, one of the parameters in our maximization is the Bernoulli parameter $\mu_{k}$, where $k$ is the index of the "mixture component", ...
0
votes
0answers
126 views

Expectation Maximization, dice example, always converging in second iteration

I am simulating two loaded dice and trying to estimate individual die prior probabilities and probability mass functions for each of them using the EM algorithm. Below is my Matlab code. Likelihood ...
1
vote
0answers
28 views

High dimensional model estimation with outliers

I have a set H of k m-dimensional hyperplanes in n dimensional space, where ...
0
votes
1answer
33 views

non-EM algorithm approach to mixture model?

I have a mixture model and the components are further parameterized by ~200 variables. Originally I use EM-algorithm to get a MLE estimation of the parameters. The algorithm works quite well and ...
0
votes
0answers
45 views

How to separate distributions from weighted dataset?

I’m trying to separate two component distributions of an apparent finite mixture from a weighted dataset (determined by a weighted.histogram). I've a set of data and weights only for a part of the ...
1
vote
0answers
102 views

How to normalize bimodal (or multimodal) distributions?

If I have multiple data series, a = [a1, a2, ... a100] ~ bimodal with mu_a1, mu_a2, sigma_a1, sigma_a2, b = [b1, b2, ... b100] ~ bimodal with mu_b1, mu_b2, sigma_b1, sigma_b2, c = [c1, c2, ... ...
0
votes
0answers
30 views

How to treat “Missing Value Analysis” test results (problems)

I have a problem with Missing Value Analysis. I am using SPSS version 20. I am trying to test whether missing values are at complete random. As I know in order to ensure missing values are completely ...
6
votes
1answer
113 views

Why does the EM algorithm have to be iterative?

Suppose that you have a population with $N$ units, each with a random variable $X_i \sim \text{Poisson}(\lambda)$. You observe $n = N-n_0$ values for any unit for which $X_i > 0$. We want an ...
0
votes
0answers
20 views

A framework for comparing the performance of Expectation Maximization

I have my own implementation of the Expectation Maximization (EM) algorithm based on the following paper http://pdf.aminer.org/000/221/588/fuzzy_k_means_clustering_with_crisp_regions.pdf I would like ...
0
votes
0answers
56 views

Estimation parameters for latent (unobserved) variable

Here is my problem: I have 3 variables $X,Y,Z$ : $X$ is the number of clicks we observed on an web advertisement; $Y$ is the number of time a customer do a sign-up on the website after clicking ...
3
votes
1answer
96 views

Relation between variational Bayes and EM

I read somewhere that Variational Bayes method is a generalization of the EM algorithm. Indeed, the iterative parts of the algorithms are very similar. In order to test whether the EM algorithm is a ...
1
vote
1answer
65 views

Relation between Gaussian mixture models and maximum likelihood?

I need some help understanding the relation between the maximum likelihood and Gaussian mixture models. I have seen that there is a relationship between the expectation maximization algorithms and ...
0
votes
0answers
31 views

Expectation maximization with variant length at observing data

Imagine one loaded dice. Based on EM algorithm, how could we compute how much it loaded if we introduced: Variant length on each rolling attempt (look at first and second attempt below 1st one has 6 ...
1
vote
0answers
30 views

Principled approach for PCA on correlated variables?

Related to Should one remove highly correlated variables before doing PCA?, PCA is used a lot in population genetics to essentially cluster individuals into ethnic group based on their genetic markers ...
2
votes
1answer
35 views

Gaussian clusters and original distributions

In Gaussian clustering (i.e. General Mixture Models) we model the data with some clusters. For example, in the below figure, we have two clusters $C_1, C_2$, each of which are modeled with a Gaussian ...
1
vote
0answers
34 views

EM Algorithm - Expectation w.r.t. a subset of current parameters

Suppose I want to make inference on a parameter vector $\theta $=$(\theta_{1},\theta_{2},\theta_{3})$ and I have some missing data $Y_{mis}$. I would like to use the EM algorithm to find the mode of ...
1
vote
0answers
50 views

Concavity of log likelihood for hidden markov models

Could you give me a good link where the concept of concavity of the log likelihood related to hidden markov model EM algorithm is clarified? Thank you in advance.
1
vote
0answers
33 views

What is delta in the maximization step in this EM algorithm?

The algorithm is used to classify english vs non-english tweets from unlabeled data. Given n observed tweets (x1 ... xn) where each tweet xi is a collection of d words (xi1 ... xid). y is the class ...
0
votes
0answers
44 views

peak detection - points / sigma ratio

I would like to ask you (statistics experts) if the following approach is kosher or a nonsense. Problem My EM based detection algorithm ends up (for some data) with a result that looks like attached ...
0
votes
1answer
76 views

Bayes' theorem in 1-d EM algorithm

I'm watching a video on the EM algorithm, When we use Bayes' Theorem to calculate $b_i$, how do I find $P(b)$ and $P(a)$ initially? It says we can estimate the priors $P(b)$ and $P(a)$ but that's ...
4
votes
2answers
123 views

EM algorithm Practice Problem

This is a practice problem for a midterm exam. The problem is an EM algorithm example. I am having trouble with part (f). I list parts (a)-(e) for completion and in case I made a mistake earlier. Let ...
3
votes
1answer
57 views

EM algorithm decreases!

I have used the Bayes Net Toolbox to build a small network, which consists of 3 nodes and is shown below. Node 1 is a Bernoulli random variable, node 2 is a Gaussian random variable and node 3 is a ...
1
vote
1answer
50 views

Why Baum-Welch algorithm is an instantiation of EM algorithm?

$\newcommand{\E}{\mathrm{E}}$ I don't understand why Baum-Welch algorithm is an instantiation of EM algorithm. Indeed, why computing $\alpha_t(i)$ and $\beta_t(i)$ corresponds to Expectation step. ...
1
vote
0answers
29 views

Mode of Joint Posterior - Maximization Problems

I have a problem whereby I get two different answers if I try to maximize a function. let $ \begin{bmatrix} Y_{o}\\ Y_{a} \end{bmatrix}|\phi\sim N (0,\phi^{-1}A^{-1}) $ $\pi(\phi)=\frac{1}{\phi}$, ...
5
votes
1answer
144 views

Why is optimizing a mixture of Gaussian directly computationally hard?

Consider the log likelihood of a mixture of guassians: $$l(S_n; \theta) = \sum^n_{t=1}logP(x^{(t)}|\theta) = \sum^n_{t=1}log\sum^k_{i=1}p_iP(x^{(t)}|\mu^{(i)}, \sigma^2_i)$$ I was wondering why it ...
0
votes
0answers
19 views

Maximum likelihood hypothesis vs Expectation maximization

Maximum Likelihood is given by the formula $h_{ML}=arg\space max_{h \in H } \space\space p(D/h)$ I want to transform it in terms of mechanism involved in Expectation maximization. $h_{ML}=arg\space ...
1
vote
0answers
22 views

Possibilities of accelerating EM algortihm

I'm trying to use the EM to estimate some parameters. I've programmed and it delivers. The problem however is that for each run of my programme, it can take either 5 seconds, 1min, 3min or more to ...
1
vote
1answer
99 views

In EM derivation why can I sum over the iid variables in the conditional expectation?

In EM when you take the expectation: $E[\log P(y,x \mid \theta)\mid x, \theta']$ $= \sum\limits_yP(y\mid x, \theta') \log P(y,x\mid \theta)$ I understand this but the following part I don't ...