Questions tagged [expectation-maximization]

An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.

Filter by
Sorted by
Tagged with
1
vote
0answers
15 views

Probabilty estimation for Bernoulli with number of trials as random variable

Problem description Suppose we have fixed number of people that are the test population, let's say $t=200$ persons. For each one of them $\mathbf{r}_j$ we know about $m=300$ features that describes ...
0
votes
0answers
17 views

Given data of 6 flips from 3 coins that obey a mixed Bernoulli distribution, what is calculation for responsibilities?

So the definition for Responsibilities is: Then the problem is defined as: So the reason I'm confused how to start is cause for the responsibility equation above $\tau(z_{nk})$ refers to parameters $...
0
votes
0answers
16 views

Local quadratic approximation

I am busy working my way through a paper by Guo et al. (pairwise variable selection for high dimensional model-based clustering) and I am just completely stuck. In the paper they use the EM algorithm ...
0
votes
0answers
10 views

Initializing EM algortihm with kmean when means are the same

I have a set of point (in one dimension) of 2 types: -First type of point: generated with gaussian density with parameters (m1,sigma1) -Second type of point: generated with gaussian density with ...
0
votes
0answers
18 views

Quadratic programming for EM algorithm

I am currently working my way through the following paper and I am stuck. Please can somebody explain to me how on earth the quadratic programming would work for the following equation? (Eq (15)-(18) ...
0
votes
0answers
12 views

Reward Function for Maximizing Distance Given Limited Amount of Power

My problem is framed as maximizing distance given a limited amount of power. Say you have a (limited) battery-powered race car that could automatically thrust its engine. You can create a function to ...
0
votes
0answers
50 views

E-Step in EM algorithm with multiple latent variables

Within EM, we conduct the E-step in order to marginalise out parameters which we can view as ‘missing’ to then find easier modal estimates of parameters of interest. Suppose the parameters to be ...
0
votes
0answers
16 views

Computationally tractable (pseudo-)likelihood methods for Gaussian data with missing values (MCAR)

I want to calculate a Gaussian (pseudo-)likelihood for some data where each data point has random dimensions missing. Calculating the marginal log-likelihood is computationally intractable because for ...
1
vote
2answers
65 views

Proportional Hazards Model and EM Algorithm

I am working with the standard proportional hazards model given by $\lambda(t|Z) = \lambda_0(t)e^{Z\beta}$ for a special type of data that requires an EM algorithm to estimate a discretized version of ...
0
votes
0answers
11 views

Differentiating expected prediction error (EPE)

From Hastie-Tibshirani-Friedman p.18-19 $$ EPE(f)=E(Y-f(x))^{2} = \int[y-f(x)]^{2}Pr(dx,dy) $$ If $f(x)\approx x^{T}\beta$ shows that by plugging in $f(x)$ in $EPE$ and differentiating w.r.t. $\beta$ ...
0
votes
0answers
21 views

Training a layered HMM

I am currently planning on training a layered Hidden Markov Model. I have 3 stages with the following structures. The first stage is a 3-state HMM with the State X: can emit insertion errors State Y: ...
0
votes
0answers
48 views

How to derive EM algorithm for this regression model?

Given that $y = Xw + \epsilon$ where $y $ is $n\times 1 $, $w$ is $p\times 1$ vector and $X$ is $n \times p$ matrix of inputs $x_1, x_2, ... x_n$ each of which are $p\times 1$ vectors. Further $\...
1
vote
1answer
77 views

EM algorithm for normal mixtures with constraints

I have $G$ groups, each with $N_g$ data points $y_{ig}$, $g=1,\dots,G$ and $i = 1, \dots, N_g$. The group for each data point in observable. I want to estimate the normal mixtures model with $K$ ...
1
vote
1answer
13 views

Why would split observed $x$ into two unobserved r.v $z_1,z_2$ consider a way to augmenting data in EM algorithm?

I am reading the materials on the EM algorithm, and I am a bit confused about the example provided on the material I am currently reading. The example is considered a classical missing data problem ...
0
votes
0answers
13 views

normal mean variance mixture garch with OBSERVED mixing variable

I need to estimate this univariate garch model with the following discription My model (regression): Yt=mu + gamma * Gt + et Gt is GIVEN Where the crucial part is that: et= sqrt(Gt) * sqrt(ht) * Zt ...
0
votes
0answers
42 views

Log-likelihood decrease in EM - two regressions

I am facing a problem with an EM algorithm where, in some iterations, the log-likelihood decreases. I have a two dimensional dataset $\{\vec{x}, \vec{y}\}$ and each data point belongs to one of these ...
0
votes
0answers
18 views

Major discrepancy of latent variable in the Gaussian Mixture Model/Expectation and Maximization literature

I have read a couple of references on the interpretation of a latent variable in the GMM/EM literature and I found a massive discrepancy between the authors so much so I now have no idea how GMM/EM ...
0
votes
1answer
89 views

About the derivation of EM for mixture of Gaussians

I'm reading Andrew Ng's note about Mixtures of Gaussians and the EM algorithm He writes the likelihood of data as where random variables $z^{(i)}$'s indicate which of the $k$ Gaussians each $x^{(i)}$ ...
0
votes
0answers
37 views

decomposition of joint density of multiple variables

I'm considering an EM algorithm of correlated random effects model $y_{it} = \beta x_{it} + \mu_i + \varepsilon_{it},(i=1,...,n;t=1,...,T)$ where $y_{it}$ and $x_{it}$ are observed, but $\mu_i$ and $\...
0
votes
0answers
31 views

Consistency of a hard EM type approach to dealing with latent variables (in a particular setting)

Suppose we observe a sample from the $d$ dimensional random vectors $Y_{1,t},\dots,Y_{N,t}$ for $t=1,\dots,T$. Suppose further that the data generating process (DGP) is the following, for $t=1,\dots,T$...
1
vote
1answer
88 views

EM Algorithm Negative Log Likelihood

I am fitting a shared parameter model where I have a binary $Y_2$ and a continuous $Y_1$, and I am using random effects in the mean structure to model them jointly $$f(Y_1,Y_2)=\int f(Y_1,Y_2,b)\,\...
0
votes
0answers
15 views

Expectation-Maximization (EM) algorithm with known means

I am trying to fit a mixture of 1D Gaussian distributions to some data. Can the EM algorithm be used in the case of known mean (all the mean values are equal to zero) and fit only the variances of the ...
4
votes
0answers
39 views

Why is the EM algorithm well suited for exponential families?

I've been brushing up on the EM algorithm, and while I feel like I understand the basics, I keep seeing the claim made (e.g. here, here, among several others) that EM works particularly well for ...
0
votes
0answers
33 views

Estimation of joint modeling of binary and continuous random variables via shared parameter

I am trying to fit a joint model of a binary and continuous outcome with repeated measures. I am trying to fit it using a shared parameter model that induces all the correlation between outcome and ...
1
vote
0answers
67 views

What calculation is implied by this Gaussian term?

I'd appreciate this community's help in understanding the calculation implied by the Gaussian term in this equation: $$w_{ik} = \alpha_{k} \mathcal{N}( \mathbf{y}_i | \mathbf{X}_i \mathbf{\beta}_k , \...
0
votes
0answers
73 views

Imputation methods for time series data (non-stationary)

I am looking for an impute method for non-stationary time series (financial indeces). From https://pypi.org/project/impyute/ (https://buildmedia.readthedocs.org/media/pdf/impyute/latest/impyute.pdf) I ...
0
votes
0answers
19 views

EM Algorithm For Distributions That Share Parameters

I've implemented an EM algorithm for a gaussian mixture where the mean is defined by an Ornstein–Uhlenbeck process. The pdf I'm trying to maximize is: $$f(r(s)|r(t)) = (1-q)\cdot e^{-(r(s) - r(t) - k(\...
0
votes
1answer
73 views

Find a lower bound for the minimax risk $\displaystyle \min_{\delta}\max_{\theta\in\Omega}EL(\theta,\delta(X))\ge1-\frac{1}{2^n}$

Consider a random binary vector $X\in\{0,1\}^n. $Let $\theta\in\Omega$ be a probability vector in $\mathbb R^{2^n}$ with $X\sim\theta$. Consider the loss function $\displaystyle L(\theta,a)=\max_{x\in\...
2
votes
0answers
54 views

Optimizing this log-likelihood

I have a HMM which emits an observation Z. The parameters of the HMM are $\boldsymbol\theta$. $$\boldsymbol\theta = {\boldsymbol{A},\boldsymbol{B},\pi}$$ Where $\boldsymbol{A}$ is the transition ...
3
votes
2answers
285 views

EM algorithm for MLE from a bivariate normal sample with missing data: Stuck on M-step

I'm trying to understand applying the EM algorithm to compute the MLE in a missing data problem. Specifically, suppose $(x_1,y_1),\ldots,(x_n,y_n)$ is a random sample from the bivariate normal ...
0
votes
0answers
29 views

Multiple imputation using proc mi EM method in SAS

I'd like to clarify how to use SAS for (multiple) imputation in SAS, specifically the EM method option for proc mi. Do I need to analyze multiple imputed sets using proc mianalyze as mentioned in ...
3
votes
1answer
118 views

Bayesian Regression- Expectation Maximization

In Bayesian regression, we have $y_i=x_i^{T}w+\epsilon_i$ where $w \sim \mathcal{N}(0,\alpha)$ and $\epsilon_i \sim \mathcal{N}(0,\frac{1}{\beta})$. Inference of $\alpha$ and $\beta$ is done by ...
0
votes
0answers
15 views

Statistical Interpretation of 2 data set measures

I was adviced to post here (initialy post on physics exchange but I am going to remove it). I have two independant experiments have measured $\tau_{1},\sigma_{1}$ and $\tau_{2},\sigma_{2}$ with $\...
0
votes
0answers
38 views

Likelihood of a latent graphical model

What is the approach to take when trying to find the likelihood of the observations on a latent graphical model, with intertwining conditional distributions? The model: Each vertex $X_i$ of a binary ...
0
votes
0answers
25 views

Derive expectation of the log determinant of a precision matrix from a Wishart distribution

I'm reading through section 21.6 of Murphy's Machine Learning: A probabilistic perspective where they derive the variational bayes algorithm for fitting a mixture of gaussians. One of the steps ...
0
votes
0answers
20 views

GMM for nonlinear mean

In conventional GMM, observations $\mathbf{X} = \left\lbrace \mathbf{x}_1,\mathbf{x}_2,\ldots\right\rbrace$ are draw from a distribution $$ \mathbf{x}_n \sim \sum_{k=1}^{K}\pi_k\mathcal{N}\left( \...
1
vote
0answers
33 views

Expectation Maximization Correctness of Problem Formulation

Suppose I draw $n$ iid samples from a Poisson$(\lambda)$ distribution, with $\lambda$ unknown. Now, I artificially turn every 3 I draw into a 1, so that the probability of observing any particular non-...
5
votes
2answers
231 views

Why do we say EM is a partially non-Bayesian method?

I have some difficulties understanding the phrase 'EM is a partially non-Bayesian method'. EM works in iterative fashion. Is it because the iterative nature of EM is somehow similar to prior - ...
0
votes
0answers
30 views

What is the nature of $r_i^t$?

Using Expectation Maximization (EM) algorithm, I want to vary the number of clusters used according to $K = [2,4, ... 50]$ for a normal distribution initialized randomly (...
0
votes
1answer
46 views

Conjectures regarding EM approximations of mixtures of multivariate normal distributions

Consider $X\in\mathbb{R}^{N\times d}$ containing data for $N$ points in $d$ dimensions drawn from a bimodal multivariate normal distribution, where any row $x$ of $X$ follows the mixed multivariate ...
2
votes
0answers
92 views

How does maximising ELBO for a Gaussian mixture model fit the model to data?

I am following along in Bishop's Pattern Recognition and ML chapters 9 and 10, and I understand that the EM algorithm works by iteratively updating model parameters using equations derived from ...
0
votes
0answers
17 views

Making sense of the belief propagation on graphs

I sort of understand when do I use variational Bayesian and when do I use expectation maximization. But now I want to know when do I use belief propagation in graphs to solve an estimation problem. ...
2
votes
1answer
58 views

EM Algorithm For Bipolar Normal Distribution

Question: Let $x_1, \dots, x_m$ be an i.i.d. sample from a normal density with mean $\mu$ and variance $\sigma^2$. Suppose for each $x_i$ we observe $y_i = |x_i|$ . Formulate an EM algorithm for ...
2
votes
1answer
48 views

Nonparametric mixture estimation

Let's assume that we have two samples $\{X_i\}_{i=1..N}$ and $\{Y_i\}_{i=1..M}$ corresponding to random variables $X$ and $Y$. Let there also be a sample $\{Z_i\}_{i=1..K}$ corresponding to random ...
1
vote
0answers
136 views

Calculating ELBO in EM algorithm

In Andrew Ng's CS229 notes, a nice derivation of EM algorithm is given. With some minor notation modifications, the algorithm is written as follows: E-step: For each i, $Q_i(z_i)=p(z_i|x_i, \theta^{...
6
votes
1answer
182 views

Finding the Q function for the EM algorithm

I have a situation where $X_1,...X_n$ come from $N(\mu,1)$ and there is a realization of 10 $x$ values. I want to use the EM algorithm to work out the MLE. So, I am trying to compute the expected ...
2
votes
1answer
131 views

Why maximizing the expected value of log likelihood under the posterior distribution of latent variables maximize the observed data log-likelihood?

I am trying to understand the Expectation-Maximization algorithm and I am not able to get the intuition of a particular step. I am able to verify the mathematical derivation but I want to understand ...
0
votes
0answers
22 views

Fixing the parameters of the variational distribution in Expectation-Maximization

Consider directed graphical model $z \to x$ (with $z$ unobserved and $x$ observed). The evidence lower bound on the log-likelihood $\log p(x) = \log \sum_z p(x, z; \theta)$ for parameters $\theta$ (...
1
vote
1answer
158 views

How to get number of iterations in EM-algorithm using R mclust gaussian mixture model

I am clustering data using the mclust function from the R mclust package. I am struggling to get the number of iterations the EM ...
2
votes
1answer
109 views

derivation of E step in EM algorithm for pLSA via Lagrangian

I have trouble deriving the EM algorithm for the Probabilistic latent semantic analysis (pLSA) model via Lagrange multipliers. I model the missing data $Q_{zij} \in \{0,1\}$ for word $w_j$ in document ...

1
2 3 4 5
11