Questions tagged [expectation-maximization]
An optimization algorithm often used for maximum-likelihood estimation in the presence of missing data.
533
questions
1
vote
0answers
15 views
Probabilty estimation for Bernoulli with number of trials as random variable
Problem description
Suppose we have fixed number of people that are the test population, let's say $t=200$ persons. For each one of them $\mathbf{r}_j$ we know about $m=300$ features that describes ...
0
votes
0answers
17 views
Given data of 6 flips from 3 coins that obey a mixed Bernoulli distribution, what is calculation for responsibilities?
So the definition for Responsibilities is:
Then the problem is defined as:
So the reason I'm confused how to start is cause for the responsibility equation above $\tau(z_{nk})$ refers to parameters $...
0
votes
0answers
16 views
Local quadratic approximation
I am busy working my way through a paper by Guo et al. (pairwise variable selection for high dimensional model-based clustering) and I am just completely stuck.
In the paper they use the EM algorithm ...
0
votes
0answers
10 views
Initializing EM algortihm with kmean when means are the same
I have a set of point (in one dimension) of 2 types:
-First type of point: generated with gaussian density with parameters (m1,sigma1)
-Second type of point: generated with gaussian density with ...
0
votes
0answers
18 views
Quadratic programming for EM algorithm
I am currently working my way through the following paper and I am stuck.
Please can somebody explain to me how on earth the quadratic programming would work for the following equation? (Eq (15)-(18) ...
0
votes
0answers
12 views
Reward Function for Maximizing Distance Given Limited Amount of Power
My problem is framed as maximizing distance given a limited amount of power. Say you have a (limited) battery-powered race car that could automatically thrust its engine.
You can create a function to ...
0
votes
0answers
50 views
E-Step in EM algorithm with multiple latent variables
Within EM, we conduct the E-step in order to marginalise out parameters which we can view as ‘missing’ to then find easier modal estimates of parameters of interest. Suppose the parameters to be ...
0
votes
0answers
16 views
Computationally tractable (pseudo-)likelihood methods for Gaussian data with missing values (MCAR)
I want to calculate a Gaussian (pseudo-)likelihood for some data where each data point has random dimensions missing. Calculating the marginal log-likelihood is computationally intractable because for ...
1
vote
2answers
65 views
Proportional Hazards Model and EM Algorithm
I am working with the standard proportional hazards model given by $\lambda(t|Z) = \lambda_0(t)e^{Z\beta}$ for a special type of data that requires an EM algorithm to estimate a discretized version of ...
0
votes
0answers
11 views
Differentiating expected prediction error (EPE)
From Hastie-Tibshirani-Friedman p.18-19
$$
EPE(f)=E(Y-f(x))^{2} = \int[y-f(x)]^{2}Pr(dx,dy)
$$
If $f(x)\approx x^{T}\beta$ shows that by plugging in $f(x)$ in $EPE$ and differentiating w.r.t. $\beta$ ...
0
votes
0answers
21 views
Training a layered HMM
I am currently planning on training a layered Hidden Markov Model.
I have 3 stages with the following structures.
The first stage is a 3-state HMM with the
State X: can emit insertion errors
State Y: ...
0
votes
0answers
48 views
How to derive EM algorithm for this regression model?
Given that $y = Xw + \epsilon$ where $y $ is $n\times 1 $, $w$ is $p\times 1$ vector and $X$ is $n \times p$ matrix of inputs $x_1, x_2, ... x_n$ each of which are $p\times 1$ vectors. Further $\...
1
vote
1answer
77 views
EM algorithm for normal mixtures with constraints
I have $G$ groups, each with $N_g$ data points $y_{ig}$, $g=1,\dots,G$ and $i = 1, \dots, N_g$. The group for each data point in observable.
I want to estimate the normal mixtures model with $K$ ...
1
vote
1answer
13 views
Why would split observed $x$ into two unobserved r.v $z_1,z_2$ consider a way to augmenting data in EM algorithm?
I am reading the materials on the EM algorithm, and I am a bit confused about the example provided on the material I am currently reading. The example is considered a classical missing data problem ...
0
votes
0answers
13 views
normal mean variance mixture garch with OBSERVED mixing variable
I need to estimate this univariate garch model with the following discription
My model (regression):
Yt=mu + gamma * Gt + et
Gt is GIVEN
Where the crucial part is that:
et= sqrt(Gt) * sqrt(ht) * Zt
...
0
votes
0answers
42 views
Log-likelihood decrease in EM - two regressions
I am facing a problem with an EM algorithm where, in some iterations, the log-likelihood decreases.
I have a two dimensional dataset $\{\vec{x}, \vec{y}\}$ and each data point belongs to one of these ...
0
votes
0answers
18 views
Major discrepancy of latent variable in the Gaussian Mixture Model/Expectation and Maximization literature
I have read a couple of references on the interpretation of a latent variable in the GMM/EM literature and I found a massive discrepancy between the authors so much so I now have no idea how GMM/EM ...
0
votes
1answer
89 views
About the derivation of EM for mixture of Gaussians
I'm reading Andrew Ng's note about Mixtures of Gaussians and the EM algorithm
He writes the likelihood of data as
where random variables $z^{(i)}$'s indicate which of the $k$ Gaussians each $x^{(i)}$ ...
0
votes
0answers
37 views
decomposition of joint density of multiple variables
I'm considering an EM algorithm of correlated random effects model
$y_{it} = \beta x_{it} + \mu_i + \varepsilon_{it},(i=1,...,n;t=1,...,T)$
where $y_{it}$ and $x_{it}$ are observed, but $\mu_i$ and $\...
0
votes
0answers
31 views
Consistency of a hard EM type approach to dealing with latent variables (in a particular setting)
Suppose we observe a sample from the $d$ dimensional random vectors $Y_{1,t},\dots,Y_{N,t}$ for $t=1,\dots,T$.
Suppose further that the data generating process (DGP) is the following, for $t=1,\dots,T$...
1
vote
1answer
88 views
EM Algorithm Negative Log Likelihood
I am fitting a shared parameter model where I have a binary $Y_2$ and a continuous $Y_1$, and I am using random effects in the mean structure to model them jointly
$$f(Y_1,Y_2)=\int f(Y_1,Y_2,b)\,\...
0
votes
0answers
15 views
Expectation-Maximization (EM) algorithm with known means
I am trying to fit a mixture of 1D Gaussian distributions to some data. Can the EM algorithm be used in the case of known mean (all the mean values are equal to zero) and fit only the variances of the ...
4
votes
0answers
39 views
Why is the EM algorithm well suited for exponential families?
I've been brushing up on the EM algorithm, and while I feel like I understand the basics, I keep seeing the claim made (e.g. here, here, among several others) that EM works particularly well for ...
0
votes
0answers
33 views
Estimation of joint modeling of binary and continuous random variables via shared parameter
I am trying to fit a joint model of a binary and continuous outcome with repeated measures. I am trying to fit it using a shared parameter model that induces all the correlation between outcome and ...
1
vote
0answers
67 views
What calculation is implied by this Gaussian term?
I'd appreciate this community's help in understanding the calculation implied by the Gaussian term in this equation:
$$w_{ik} = \alpha_{k} \mathcal{N}( \mathbf{y}_i | \mathbf{X}_i \mathbf{\beta}_k , \...
0
votes
0answers
73 views
Imputation methods for time series data (non-stationary)
I am looking for an impute method for non-stationary time series (financial indeces).
From https://pypi.org/project/impyute/ (https://buildmedia.readthedocs.org/media/pdf/impyute/latest/impyute.pdf) I ...
0
votes
0answers
19 views
EM Algorithm For Distributions That Share Parameters
I've implemented an EM algorithm for a gaussian mixture where the mean is defined by an Ornstein–Uhlenbeck process. The pdf I'm trying to maximize is:
$$f(r(s)|r(t)) = (1-q)\cdot e^{-(r(s) - r(t) - k(\...
0
votes
1answer
73 views
Find a lower bound for the minimax risk $\displaystyle \min_{\delta}\max_{\theta\in\Omega}EL(\theta,\delta(X))\ge1-\frac{1}{2^n}$
Consider a random binary vector $X\in\{0,1\}^n. $Let $\theta\in\Omega$ be a probability vector in $\mathbb R^{2^n}$ with $X\sim\theta$. Consider the loss function $\displaystyle L(\theta,a)=\max_{x\in\...
2
votes
0answers
54 views
Optimizing this log-likelihood
I have a HMM which emits an observation Z.
The parameters of the HMM are $\boldsymbol\theta$. $$\boldsymbol\theta = {\boldsymbol{A},\boldsymbol{B},\pi}$$
Where $\boldsymbol{A}$ is the transition ...
3
votes
2answers
285 views
EM algorithm for MLE from a bivariate normal sample with missing data: Stuck on M-step
I'm trying to understand applying the EM algorithm to compute the MLE in a missing data problem.
Specifically, suppose $(x_1,y_1),\ldots,(x_n,y_n)$ is a random sample from the bivariate normal ...
0
votes
0answers
29 views
Multiple imputation using proc mi EM method in SAS
I'd like to clarify how to use SAS for (multiple) imputation in SAS, specifically the EM method option for proc mi. Do I need to analyze multiple imputed sets using proc mianalyze as mentioned in ...
3
votes
1answer
118 views
Bayesian Regression- Expectation Maximization
In Bayesian regression, we have $y_i=x_i^{T}w+\epsilon_i$ where $w \sim \mathcal{N}(0,\alpha)$ and $\epsilon_i \sim \mathcal{N}(0,\frac{1}{\beta})$.
Inference of $\alpha$ and $\beta$ is done by ...
0
votes
0answers
15 views
Statistical Interpretation of 2 data set measures
I was adviced to post here (initialy post on physics exchange but I am going to remove it).
I have two independant experiments have measured $\tau_{1},\sigma_{1}$ and $\tau_{2},\sigma_{2}$ with $\...
0
votes
0answers
38 views
Likelihood of a latent graphical model
What is the approach to take when trying to find the likelihood of the observations on a latent graphical model, with intertwining conditional distributions?
The model:
Each vertex $X_i$ of a binary ...
0
votes
0answers
25 views
Derive expectation of the log determinant of a precision matrix from a Wishart distribution
I'm reading through section 21.6 of Murphy's Machine Learning: A probabilistic perspective where they derive the variational bayes algorithm for fitting a mixture of gaussians. One of the steps ...
0
votes
0answers
20 views
GMM for nonlinear mean
In conventional GMM, observations $\mathbf{X} = \left\lbrace \mathbf{x}_1,\mathbf{x}_2,\ldots\right\rbrace$ are draw from a distribution
$$
\mathbf{x}_n \sim \sum_{k=1}^{K}\pi_k\mathcal{N}\left( \...
1
vote
0answers
33 views
Expectation Maximization Correctness of Problem Formulation
Suppose I draw $n$ iid samples from a Poisson$(\lambda)$ distribution, with $\lambda$ unknown. Now, I artificially turn every 3 I draw into a 1, so that the probability of observing any particular non-...
5
votes
2answers
231 views
Why do we say EM is a partially non-Bayesian method?
I have some difficulties understanding the phrase 'EM is a partially non-Bayesian method'. EM works in iterative fashion. Is it because the iterative nature of EM is somehow similar to prior - ...
0
votes
0answers
30 views
What is the nature of $r_i^t$?
Using Expectation Maximization (EM) algorithm, I want to vary the number of clusters used according to $K = [2,4, ... 50]$ for a normal distribution initialized randomly (...
0
votes
1answer
46 views
Conjectures regarding EM approximations of mixtures of multivariate normal distributions
Consider $X\in\mathbb{R}^{N\times d}$ containing data for $N$ points in $d$ dimensions drawn from a bimodal multivariate normal distribution, where any row $x$ of $X$ follows the mixed multivariate ...
2
votes
0answers
92 views
How does maximising ELBO for a Gaussian mixture model fit the model to data?
I am following along in Bishop's Pattern Recognition and ML chapters 9 and 10, and I understand that the EM algorithm works by iteratively updating model parameters using equations derived from ...
0
votes
0answers
17 views
Making sense of the belief propagation on graphs
I sort of understand when do I use variational Bayesian and when do I use expectation maximization. But now I want to know when do I use belief propagation in graphs to solve an estimation problem.
...
2
votes
1answer
58 views
EM Algorithm For Bipolar Normal Distribution
Question:
Let $x_1, \dots, x_m$ be an i.i.d. sample from a normal density with mean $\mu$ and variance $\sigma^2$.
Suppose for each $x_i$ we observe $y_i = |x_i|$ . Formulate an EM algorithm for
...
2
votes
1answer
48 views
Nonparametric mixture estimation
Let's assume that we have two samples $\{X_i\}_{i=1..N}$ and $\{Y_i\}_{i=1..M}$ corresponding to random variables $X$ and $Y$. Let there also be a sample $\{Z_i\}_{i=1..K}$ corresponding to random ...
1
vote
0answers
136 views
Calculating ELBO in EM algorithm
In Andrew Ng's CS229 notes, a nice derivation of EM algorithm is given. With some minor notation modifications, the algorithm is written as follows:
E-step: For each i, $Q_i(z_i)=p(z_i|x_i, \theta^{...
6
votes
1answer
182 views
Finding the Q function for the EM algorithm
I have a situation where $X_1,...X_n$ come from $N(\mu,1)$ and there is a realization of 10 $x$ values. I want to use the EM algorithm to work out the MLE.
So, I am trying to compute the expected ...
2
votes
1answer
131 views
Why maximizing the expected value of log likelihood under the posterior distribution of latent variables maximize the observed data log-likelihood?
I am trying to understand the Expectation-Maximization algorithm and I am not able to get the intuition of a particular step. I am able to verify the mathematical derivation but I want to understand ...
0
votes
0answers
22 views
Fixing the parameters of the variational distribution in Expectation-Maximization
Consider directed graphical model $z \to x$ (with $z$ unobserved and $x$ observed). The evidence lower bound on the log-likelihood $\log p(x) = \log \sum_z p(x, z; \theta)$ for parameters $\theta$ (...
1
vote
1answer
158 views
How to get number of iterations in EM-algorithm using R mclust gaussian mixture model
I am clustering data using the mclust function from the R mclust package. I am struggling to get the number of iterations the EM ...
2
votes
1answer
109 views
derivation of E step in EM algorithm for pLSA via Lagrangian
I have trouble deriving the EM algorithm for the Probabilistic latent semantic analysis (pLSA) model via Lagrange multipliers.
I model the missing data $Q_{zij} \in \{0,1\}$ for word $w_j$ in document ...
