Questions tagged [expectation-maximization]

An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.

Filter by
Sorted by
Tagged with
0 votes
0 answers
14 views

Hidden Markov Model - Baum Welch algorithm initialization

Currently I'm working on a problem where I have a multidimensional, continuous sequence of observations $X$ that model my response variable $y$ with two states $0$ and $1$. I assume that this sequence ...
user avatar
  • 249
0 votes
0 answers
14 views

How to conduct EM algorithm when there are some outliers in GMM Models?

I'm just confused about the problem of adding an outlier component directly to the primary form of GMM models: Suppose that the observed data contains several outliers. The mixture model could be: $$ ...
user avatar
  • 1
0 votes
1 answer
19 views

Conditional independence in EM algorithm

Let $X$, $\theta$ and $Z$ denote observed, parameter and latent nodes in a graphical model. The EM algorithm attempts to find a local maximum likelihood estimate $\theta^\ast$ for the likelihood of ...
user avatar
  • 212
3 votes
1 answer
91 views

Is the likelihood for Gaussian mixture models still multimodal when Y is partially observed?

In discussing Gaussian mixture models (GMMs), https://normaldeviate.wordpress.com/2012/08/04/mixture-models-the-twilight-zone-of-statistics/ highlights the issue of Multimodality of the Likelihood. ...
user avatar
  • 3,768
1 vote
1 answer
39 views

Issue with Casella&Berger derivation of EM likelihood equality

In the explanation of the EM (Expectation maximization) algorithm p.328 in the book "Statistical inference" by G. Casella and R. Berger, 2nd edition, they present the following: $\mathbf{Y} =...
user avatar
0 votes
0 answers
12 views

Expectation-Maximization high missing rates and multiple variables

I know MICE can be used for imputation of multiple variables simultaneously. The expectation maximization approach (EM) can be used to impute missing data. Typically, one should only be using ...
user avatar
0 votes
0 answers
18 views

Labeling unlabeled data (expectation maximization)

Say I have a database (Excel) consisting of 10k different dresses and accompanying attributes (column names) for each dress (sleeve length, color, pattern, ...). I would like to label each of these ...
user avatar
0 votes
1 answer
78 views

How does expectation maximization compute missing data?

I have been searching for a simple example of how expectation-maximization (EM) computes missing data. All the examples I have found are based on multivariate normal models. I have seen that EM can be ...
user avatar
1 vote
0 answers
46 views

Problems with convergence of the EM algorithm for a gaussian mixture regression

I have been implementing a EM-algorithm for a latent-class regression model, where every individual has a vector of observations. Currenly, I have the problem that the model does not converge. The log ...
user avatar
0 votes
0 answers
55 views

EM derivation for mixture of t distributions

(Notation is at the bottom.) I'm currently self-studying the Expectation-Maximization algorithm and has learned to derive it for various mixture models including Gaussian mixture models, mixture of ...
user avatar
  • 124
0 votes
0 answers
9 views

Model choice with expectation maximization: which likelihood?

When deciding about the number of mixture components using Akaike or bayesian information criteria, should one use the full likelihood or the likelihood marginalized over the latent variables? Both ...
user avatar
  • 1,578
0 votes
0 answers
37 views

Benefits of Expectation Maximization for Mixture Models

What are the benefits of using expectation maximization for mixture models vs. direct maximization of the marginal likelihoods? Analytic maximization step In case of Gaussian mixtures the benefit is ...
user avatar
  • 1,578
0 votes
1 answer
91 views

Computing a prior from two components in Naive Bayes

Given a model parameter $\theta$ that is composed of two distributions in a Naive Bayes classifier, how is $P(\theta)$ typically computed in practice? More specifically, from the article of Nigam et ...
user avatar
0 votes
0 answers
40 views

Expectation maximization: does the likelihood always increase monotonically?

When working with (gaussian) mixture models, I always took it for a mathematical fact that the marginal likelihood increases with every iteration step. If it were not the case, it always meant an ...
user avatar
  • 1,578
0 votes
0 answers
46 views

EM algorithm for multinomial data

Good afternoon , Assume we have a dataset modeled by multinomial data of K cluster. To apply EM algorithm parameters are : $${\theta }^{\left(0\right)}=({p}_{k}^{\left(0\right)},k=1,\mathrm{\dots },K)$...
user avatar
  • 113
2 votes
0 answers
42 views

Can the EM algorithm be used to perform model selection?

Suppose that the samples $\mathcal{D} = \{x_1,x_2,\dots,x_N\}$ were independently sampled from a Gaussian distribution with mean $\mu$ and variance $\sigma^2$. However, suppose that we did not know ...
user avatar
  • 2,916
0 votes
0 answers
10 views

Residual Diagnostics of an Approximate Dynamic Factor Model estimated through EM-Algorithm

I am estimating and forecasting macroeconomic variables using an approximate dynamic factor model using EM-Algorithm. The model allows the idiosyncratic terms to follow an AR(1) process. So in OLS ...
user avatar
  • 109
1 vote
1 answer
156 views

Derivation of the M-step in EM algorithm for a three-dimensional panel mixture model

I have a question regarding the estimation of a latent-class gaussian mixture model, where the model is for three dimensional panel data set with individuals $i$, in country $j$ in time $t$. I want ...
user avatar
0 votes
0 answers
28 views

Applying EM Algorithm to 3x3 Contingency Table

Given the contingency table $$ \begin{array}{l} \begin{array}{l|ccc} & \text { Alcholoic } & \text {Non-Alcholic} & \text { Not Answered } \\ \hline \text { Male } & n_{11} & n_{...
user avatar
  • 1
2 votes
2 answers
35 views

Practical considerations for EM clustering

EM algorithm guarantees finding a local rather then global minimum of the likelihood. As a consequence, the results are dependent on the initial conditions (e.g., if randomly choosing the initial ...
user avatar
  • 1,578
1 vote
1 answer
354 views

How can I derive the EM algorithm for a mixture of two Bernoulli distributions?

How can I derive the E-step and M-step in the EM algorithm for a mixture of two Bernoulli distributions? Note that I am aware that there are several notes online that explain how to do this for the ...
user avatar
  • 2,916
15 votes
2 answers
666 views

Finding category with maximum likelihood method

Let's say that we had an information for men and women heights. R code: ...
user avatar
4 votes
1 answer
92 views

Why use the EM Algorithm and not just marginalise the complete likelihood?

On the wikipedia article for Expectation-Maximization it states Given the statistical model which generates a set $\mathbf{X}$ of observed data, a set of unobserved latent data or missing values $\...
user avatar
0 votes
0 answers
14 views

EM (?) algorithm with dependent observations but the prior seems not informative

How can we derive the (EM?) algorithm to estimate $b$ if we suppose $G_j = bg_j$ for all $j$, the priors (?) of the observations $\hat{G}_j \sim N(\hat{G}_j | G_j, (s_{G, j})^2)$, $\hat{g}_j \sim N(\...
user avatar
  • 101
0 votes
1 answer
33 views

EM algorithm when there are too many components to calculate the function Q

Assuming a regression model as follow: $$\mathbf{y} = \mathbf{x}\beta + \mathbf{\varepsilon}$$ where $\mathbf{y}=(y_1,...,y_n)^T\in\mathbb{R}^{n\times 1}$, $\mathbf{x}=(x_1,...,x_n)^T\in\mathbb{R}^{n\...
user avatar
  • 1
1 vote
2 answers
108 views

Is this interpretation of the EM algorithm correct?

Suppose we want to estimate the parameter $\theta$ of the distribution $p(z;\theta)$ of a random variable $Z$. If we had the i.i.d samples $z_1, z_2, \dots, z_N$, we could easily do this using maximum ...
user avatar
  • 2,916
0 votes
0 answers
28 views

Maximum likelihood expectation maximization (for image segmentation)

I am trying to wrap my head around the concept of using Maximum likelihood expectation maximization for image segmentation. I understand the concept of maximum likelihood as a way of finding the most ...
user avatar
1 vote
1 answer
48 views

EM algorithm and random effects

I am trying to learn how to fit random effects/mixed effects models. I read on this article that quasi-likelihood, numerical integration and MCMC are the most common methods. My question is: can we ...
user avatar
  • 2,536
0 votes
0 answers
26 views

Reference Request: Variational Expectation-Maximization algorithm for Latent Dirichlet Allocation with an added time component

This link has a pretty good runthrough on the variational inference (via variational E-M) for LDA with calculations expanded and explained. I am now considering a modified LDA which adds a time ...
user avatar
  • 1
4 votes
1 answer
429 views

K Means as a special case of GMM (using EM Algorithm)

I am looking for a tutorial/gentle introduction (preferably with mathematics/proofs) on K-means as a special case of Gaussian Mixture Model using the EM Algorithm. I have found this: https://www....
user avatar
  • 7,545
1 vote
1 answer
69 views

Understanding numerical example of expectation maximization

I was trying to understand Expectation maximization algorithm. This is how it is defined in Andrew Ng's Stanford CS229 course: $$ \text{Repeat until convergence \{}\quad\quad\quad\quad\quad\quad\...
user avatar
  • 123
0 votes
0 answers
21 views

Expected 0-1 under multinomial distribution

Suppose the world generates a single observation $x \sim \text{multinomial}(\theta)$, where the parameter vector $\theta = (\theta_1,\ldots, \theta_k)$ with $\theta_i\geq 0$ and $\theta_1+\ldots+\...
user avatar
  • 121
2 votes
1 answer
172 views

Is there a special case of the EM algorithm for exponential family distributions?

According to Wikipedia, the formal definition of the EM algorithm is The EM algorithm seeks to find the MLE of the marginal likelihood by iteratively applying these two steps: Expectation step (E ...
user avatar
  • 2,916
0 votes
0 answers
51 views

Areas of research in statistical machine learning

I'm reading from a book called Machine Learning: A Probabilistic Perspective by Kevin Murphy. Besides being somewhat challenging to understand, I feel that the earlier chapters (on probability, ...
0 votes
1 answer
32 views

Penalized model based clustering variable selection with L1 penalty

Ok so I am working through a paper by Pan and Shen where they maximize a penalized version of the complete data log-likelihood $$ Q_{P}\left(\Theta ; \Theta^{(m)}\right)=E_{\Theta^{(m)}}\left(\log L_{...
user avatar
0 votes
0 answers
12 views

In theory, does the form of the latent dynamics matter?

I am experimenting with latent variable models of the form $\mathbf{x}_t = \mathbf{W}\mathbf{z}_t + \mathcal{E}_t$ where $\mathcal{E}_t \sim \mathcal{N}(0,\sigma^2\mathbf{I})$ and $\mathbf{z}_t \sim \...
user avatar
  • 101
0 votes
1 answer
49 views

Learning HMM parameters by counting?

In 8.4.3 of the book Speech and Language Processing: An introduction to natural language processing, the two matrices transition probabilities and emission probabilities can be learned by counting as ...
user avatar
  • 5,151
0 votes
1 answer
60 views

Jensen's inequality in EM derivation: expectation derivation

In Where does Jensen's Inequality come into the EM derivation? and in McLachlan & Krishnan (1997) - The EM Algorithm and Extensions, it was shown that \begin{equation} H(\theta|\theta^{(t)}) \...
user avatar
0 votes
0 answers
12 views

Are the subsets of Mixture Models necessarily parametrized?

Im just learning about mixture models and they are described as a "mix of parametric and non parametric models". My question is how a non-parametric subset would look like? How would e.g. EM-...
user avatar
  • 1
0 votes
1 answer
119 views

Monte Carlo approximation to find expected value of gradient square

I need to to calculate this term: $ \mathbb{E}\left[S(Y, L,\theta)S(Y,L,\theta)^\prime\right] $ Where $ S(Y,L,\theta) =\frac{\partial}{\partial\theta} l(Y,L,\theta) $ With $\theta$ = maximum ...
user avatar
0 votes
0 answers
26 views

Multivariate Normal Underflow

Good day everyone At the moment I am attempting to write code in R to calculate the following. $$ \tau_{k j}^{(m)}=\frac{\pi_{k}^{(m)} f_{k}\left(x_{j} ; \theta_{k}^{(m)}\right)}{f\left(x_{j} ; \Theta^...
user avatar
1 vote
1 answer
80 views

Expected log likelihood for mixture components with differing support

I was hoping to use the EM algorithm to fit a mixture model in which the mixture components can have differing support. I've run into a problem during the M step because the expected log-likelihood ...
user avatar
  • 13
1 vote
0 answers
27 views

High dimensional multivariate normal calculations in R

Good day everyone At the moment I am attempting to write code in R to calculate the following. $$ \tau_{k j}^{(m)}=\frac{\pi_{k}^{(m)} f_{k}\left(x_{j} ; \theta_{k}^{(m)}\right)}{f\left(x_{j} ; \Theta^...
user avatar
2 votes
1 answer
77 views

linear regression inappropriate when there are multiple underlying groups

In this scenario, one group has a clear linear relationship between x and y. Another group doesn't. However, there is no way to differentiate them in the data. In this case, performing a simple linear ...
user avatar
0 votes
0 answers
173 views

EM algorithm for multivariate gaussian with diagonal covariance matrix

Ok so quick question. Say I need to use the EM-algorithm to estimate the parameters of a multivariate gaussian $$ f_{k}\left(x ; \theta_{k}\right)=\frac{1}{(2 \pi)^{P / 2}|V|} \exp \left(-\frac{1}{2}\...
user avatar
0 votes
1 answer
74 views

What are the variations of Expectation Maximization?

To explain my question better, I will use this analogy: In the case of the Gradient-Descent method, we have multiple variations/expansions for the main algorithm, like stochastic gradient descent (SGD)...
user avatar
2 votes
2 answers
99 views

Why is $q(\mathbf{z})$ chosen to be the posterior distribution in the EM algorithm?

In the CS229 Lecture Notes on the EM algorithm by Tengyu Ma and Andrew Ng (2019), the authors write that $$ \log(p(\mathbf{x};\theta)) = \log\left(\mathbb{E}_{q(\mathbf{z})}\left[\frac{p(\mathbf{x},\...
user avatar
  • 2,916
0 votes
0 answers
74 views

EM Algorithm for Kalman Filters

Say I have the following dynamical system with unknown covariances for $w,v$. $$ z_n = Az_{n-1}+w\\ x_n = Bz_n +v $$ where $w \sim \mathcal{N}(0,Q)$ and $v \sim \mathcal{N}(0,R)$. I want to apply the ...
user avatar
  • 177
0 votes
0 answers
25 views

Why isn't this term considered an expectation in EM algorithm?

The log-likelihood $\mathcal{L}$ for a mixture model can be written as: $$ \mathcal{L} = \log p(\boldsymbol X| \Delta, \boldsymbol \pi) = \sum_{n=1}^N \log \color{blue}{\sum_{k=1}^K \pi_k p(x_n|\...
user avatar
  • 113
0 votes
1 answer
41 views

Regression where the matrix is drawn from a known distribution

I need to estimate the vector x, where Ax = b, given b is observed data and the columns in the matrix A are drawn from a known distribution. a and b are both formed of positive integers (or zero). For ...
user avatar
  • 1

1
2 3 4 5
12