Return to Question

In its general form the E-step of the EM algorithm finds the expectation

$$ Q(\theta|\theta') =\int \log[ p(Y,Z | \theta)] p(Z|Y,\theta') d Z$$

where $Y$ the data, $Z$ the latent variables, $\theta'$ the current parameters, and $l(\theta|Y,Z) = p(Y,Z|\theta)$ the complete data likelihood.

My general question is: does EM require us to know the joint conditional (predictive) distribution of the latent variables, $p(Z|Y,\theta)$?

Context. This question is phrased with more context as follows. Suppose $Z=[X,S]$ has two random variables parameterised by distinct vectors $\lambda$ and $\phi$ respectively, i.e. we know (can evaluate and sample from) $p(X,S|\lambda,\phi)=p(X|\lambda)p(S|\phi)$. For my problem, data $Y$ are a deterministic function $Y=f(X,S)$.

Variables $X,S$ are marginally independent; however they are not independent conditional on $Y$. We also know the complete data likelihood, it is

$$p(Y,X,S|\lambda, \phi) = p(X,S|\lambda, \phi) = p(X|\lambda)p(S|\phi).$$

Also we know the full conditionals $p(X|S,Y,\lambda)$ and $p(S|X,Y,\phi)$. They denote truncated distributions which can be derived. If we fill this in we find:

$$ \int \log[ p(X|\lambda)p(S|\phi) ] p(X,S|Y, \phi',\lambda') dXdS$$

As noted by Xi'an the E step then considers

$$ Q(\lambda, \phi|\lambda',\phi') = E_{X,S|Y,\lambda',\phi'}[l(\lambda|X)] +E_{X,S|Y,\lambda',\phi'}[l(\phi|S)] $$

which could be maximized in turn in an expectation conditional maximization (ECM) algorithm. However, for the problem I am working on the joint predictive distribution $p(X,S|Y, \phi,\lambda)$ is not known. We can use the (known) truncated distributions to write

$$ p(X|S,Y, \lambda) p(S|Y,\phi) = p(S|X,Y, \phi) p(X|Y,\lambda)$$

However the marginals $p(X|Y,\lambda),p(S|Y,\phi)$ are then still unknown for my problem. As Xi'an noted the log-likelihood simplifies to

$$ Q(\lambda, \phi|\lambda',\phi') = E_{X|Y,\lambda'}[l(\lambda|X)] +E_{S|Y,\phi'}[l(\phi|S)] $$

involving the same unknown densities.

Is there still a way to use a type of EM algorithm to solve this estimation problem?

Speculative procedure. Let us assume we know $S=s_0$ then we could use EM with

$$ \tilde{Q}_1(\lambda, \phi|\lambda',\phi') = E_{X|Y,s_0,\lambda'}[l(\lambda|X)] $$

Similarly if we knew $X=x_0$

$$ \tilde{Q}_2(\lambda, \phi|\lambda',\phi') = E_{S|Y,x_0,\phi'}[l(\phi|S)] $$

So I gauge that a procedure that updates as follows might be a valid EM?

1. Sample $s_0 \sim p(S|\phi')$
2. Update $\lambda'$ by maximizing $\tilde{Q}_1$
3. Sample $x_0 \sim p(X|\lambda')$
4. Update $\phi'$ by maximizing $\tilde{Q}_2$

Then iterate until convergence. This seems not pure EM as it uses data augmentation. But perhaps something like this is a known method?

In its general form the E-step of the EM algorithm finds the expectation

$$ Q(\theta|\theta') =\int \log[ p(Y,Z | \theta)] p(Z|Y,\theta') d Z$$

where $Y$ the data, $Z$ the latent variables, $\theta'$ the current parameters, and $l(\theta|Y,Z) = p(Y,Z|\theta)$ the complete data likelihood.

My general question is: does EM require us to know the joint conditional (predictive) distribution of the latent variables, $p(Z|Y,\theta)$?

Context. This question is phrased with more context as follows. Suppose $Z=[X,S]$ has two random variables parameterised by distinct vectors $\lambda$ and $\phi$ respectively, i.e. we know (can evaluate and sample from) $p(X,S|\lambda,\phi)=p(X|\lambda)p(S|\phi)$. For my problem, data $Y$ are a deterministic function $Y=f(X,S)$.

Variables $X,S$ are marginally independent; however they are not independent conditional on $Y$. We also know the complete data likelihood, it is

$$p(Y,X,S|\lambda, \phi) = p(X,S|\lambda, \phi) = p(X|\lambda)p(S|\phi).$$

Also we know the full conditionals $p(X|S,Y,\lambda)$ and $p(S|X,Y,\phi)$. They denote truncated distributions which can be derived. If we fill this in we find:

$$ \int \log[ p(X|\lambda)p(S|\phi) ] p(X,S|Y, \phi',\lambda') dXdS$$

As noted by Xi'an the E step then considers

$$ Q(\lambda, \phi|\lambda',\phi') = E_{X,S|Y,\lambda',\phi'}[l(\lambda|X)] +E_{X,S|Y,\lambda',\phi'}[l(\phi|S)] $$

which could be maximized in turn in an expectation conditional maximization (ECM) algorithm. However, for the problem I am working on the joint predictive distribution $p(X,S|Y, \phi,\lambda)$ is not known. We can use the (known) truncated distributions to write

$$ p(X|S,Y, \lambda) p(S|Y,\phi) = p(S|X,Y, \phi) p(X|Y,\lambda)$$

However the marginals $p(X|Y,\lambda),p(S|Y,\phi)$ are then unknown for my problem. As Xi'an noted the log-likelihood simplifies to

$$ Q(\lambda, \phi|\lambda',\phi') = E_{X|Y,\lambda'}[l(\lambda|X)] +E_{S|Y,\phi'}[l(\phi|S)] $$

involving the same unknown densities.

Is there still a way to use a type of EM algorithm to solve this estimation problem?

Speculative procedure. Let us assume we know $S=s_0$ then we could use EM with

$$ \tilde{Q}_1(\lambda, \phi|\lambda',\phi') = E_{X|Y,s_0,\lambda'}[l(\lambda|X)] $$

Similarly if we knew $X=x_0$

$$ \tilde{Q}_2(\lambda, \phi|\lambda',\phi') = E_{S|Y,x_0,\phi'}[l(\phi|S)] $$

So I gauge that a procedure that updates as follows might be a valid EM?

1. Sample $s_0 \sim p(S|\phi')$
2. Update $\lambda'$ by maximizing $\tilde{Q}_1$
3. Sample $x_0 \sim p(X|\lambda')$
4. Update $\phi'$ by maximizing $\tilde{Q}_2$

Then iterate until convergence. This seems not pure EM as it uses data augmentation. But perhaps something like this is a known method?

In its general form the E-step of the EM algorithm finds the expectation

$$ Q(\theta|\theta') =\int \log[ p(Y,Z | \theta)] p(Z|Y,\theta') d Z$$

where $Y$ the data, $Z$ the latent variables, $\theta'$ the current parameters, and $l(\theta|Y,Z) = p(Y,Z|\theta)$ the complete data likelihood.

My general question is: does EM require us to know the joint conditional (predictive) distribution of the latent variables, $p(Z|Y,\theta)$?

Context. This question is phrased with more context as follows. Suppose $Z=[X,S]$ has two random variables parameterised by distinct vectors $\lambda$ and $\phi$ respectively, i.e. we know (can evaluate and sample from) $p(X,S|\lambda,\phi)=p(X|\lambda)p(S|\phi)$. For my problem, data $Y$ are a deterministic function $Y=f(X,S)$.

Variables $X,S$ are marginally independent; however they are not independent conditional on $Y$. We also know the complete data likelihood, it is

$$p(Y,X,S|\lambda, \phi) = p(X,S|\lambda, \phi) = p(X|\lambda)p(S|\phi).$$

Also we know the full conditionals $p(X|S,Y,\lambda)$ and $p(S|X,Y,\phi)$. They denote truncated distributions which can be derived. If we fill this in we find:

$$ \int \log[ p(X|\lambda)p(S|\phi) ] p(X,S|Y, \phi',\lambda') dXdS$$

As noted by Xi'an the E step then considers

$$ Q(\lambda, \phi|\lambda',\phi') = E_{X,S|Y,\lambda',\phi'}[l(\lambda|X)] +E_{X,S|Y,\lambda',\phi'}[l(\phi|S)] $$

which could be maximized in turn in an expectation conditional maximization (ECM) algorithm. However, for the problem I am working on the joint predictive distribution $p(X,S|Y, \phi,\lambda)$ is not known. We can use the (known) truncated distributions to write

$$ p(X|S,Y, \lambda) p(S|Y,\phi) = p(S|X,Y, \phi) p(X|Y,\lambda)$$

However the marginals $p(X|Y,\lambda),p(S|Y,\phi)$ are then still unknown for my problem. As Xi'an noted the log-likelihood simplifies to

$$ Q(\lambda, \phi|\lambda',\phi') = E_{X|Y,\lambda'}[l(\lambda|X)] +E_{S|Y,\phi'}[l(\phi|S)] $$

involving the same unknown densities.

Is there still a way to use a type of EM algorithm to solve this estimation problem?

If, for example, I would be allowed to augment $S$ and $X$ at some point (i.e. hold them fix at a given EM-step) it seems that $Q$ would only depend on the truncated distributions which can be calculated. However I am not sure if such holding fix the latent variates is allowed in EM.

In its general form the E-step of the EM algorithm finds the expectation

$$ Q(\theta|\theta') =\int \log[ p(Y,Z | \theta)] p(Z|Y,\theta') d Z$$

where $Y$ the data, $Z$ the latent variables, $\theta'$ the current parameters, and $l(\theta|Y,Z) = p(Y,Z|\theta)$ the complete data likelihood.

My general question is: does EM require us to know the joint conditional (predictive) distribution of the latent variables, $p(Z|Y,\theta)$?

Context. This question is phrased with more context as follows. Suppose $Z=[X,S]$ has two random variables parameterised by distinct vectors $\lambda$ and $\phi$ respectively, i.e. we know (can evaluate and sample from) $p(X,S|\lambda,\phi)=p(X|\lambda)p(S|\phi)$. For my problem, data $Y$ are a deterministic function $Y=f(X,S)$.

Variables $X,S$ are marginally independent; however they are not independent conditional on $Y$. We also know the complete data likelihood, it is

$$p(Y,X,S|\lambda, \phi) = p(X,S|\lambda, \phi) = p(X|\lambda)p(S|\phi).$$

Also we know the full conditionals $p(X|S,Y,\lambda)$ and $p(S|X,Y,\phi)$. They denote truncated distributions which can be derived. If we fill this in we find:

$$ \int \log[ p(X|\lambda)p(S|\phi) ] p(X,S|Y, \phi',\lambda') dXdS$$

As noted by Xi'an the E step then considers

$$ Q(\lambda, \phi|\lambda',\phi') = E_{X,S|Y,\lambda',\phi'}[l(\lambda|X)] +E_{X,S|Y,\lambda',\phi'}[l(\phi|S)] $$

which could be maximized in turn in an expectation conditional maximization (ECM) algorithm. However, for the problem I am working on the joint predictive distribution $p(X,S|Y, \phi,\lambda)$ is not known. We can use the (known) truncated distributions to write

$$ p(X|S,Y, \lambda) p(S|Y,\phi) = p(S|X,Y, \phi) p(X|Y,\lambda)$$

However the marginals $p(X|Y,\lambda),p(S|Y,\phi)$ are then still unknown for my problem. As Xi'an noted the log-likelihood simplifies to

$$ Q(\lambda, \phi|\lambda',\phi') = E_{X|Y,\lambda'}[l(\lambda|X)] +E_{S|Y,\phi'}[l(\phi|S)] $$

involving the same unknown densities.

Is there still a way to use a type of EM algorithm to solve this estimation problem?

Speculative procedure. Let us assume we know $S=s_0$ then we could use EM with

$$ \tilde{Q}_1(\lambda, \phi|\lambda',\phi') = E_{X|Y,s_0,\lambda'}[l(\lambda|X)] $$

Similarly if we knew $X=x_0$

$$ \tilde{Q}_2(\lambda, \phi|\lambda',\phi') = E_{S|Y,x_0,\phi'}[l(\phi|S)] $$

So I gauge that a procedure that updates as follows might be a valid EM?

1. Sample $s_0 \sim p(S|\phi')$
2. Update $\lambda'$ by maximizing $\tilde{Q}_1$
3. Sample $x_0 \sim p(X|\lambda')$
4. Update $\phi'$ by maximizing $\tilde{Q}_2$

Then iterate until convergence. This seems not pure EM as it uses data augmentation. But perhaps something like this is a known method?

In its general form the E-step of the EM algorithm finds the expectation

$$ Q(\theta|\theta') =\int \log[ p(Y,Z | \theta)] p(Z|Y,\theta') d Z$$

where $Y$ the data, $Z$ the latent variables, $\theta'$ the current parameters, and $l(\theta|Y,Z) = p(Y,Z|\theta)$ the complete data likelihood.

My general question is: does EM require us to know the joint conditional (predictive) distribution of the latent variables, $p(Z|Y,\theta)$?

Context. This question is phrased with more context as follows. Suppose $Z=[X,S]$ has two random variables parameterised by distinct vectors $\lambda$ and $\phi$ respectively, i.e. we know (can evaluate and sample from) $p(X,S|\lambda,\phi)=p(X|\lambda)p(S|\phi)$. For my problem, data $Y$ are a deterministic function $Y=f(X,S)$.

Variables $X,S$ are marginally independent; however they are not independent conditional on $Y$. We also know the complete data likelihood, it is

$$p(Y,X,S|\lambda, \phi) = p(X,S|\lambda, \phi) = p(X|\lambda)p(S|\phi).$$

Also we know the full conditionals $p(X|S,Y,\lambda)$ and $p(S|X,Y,\phi)$. They denote truncated distributions which can be derived. If we fill this in we find:

$$ \int \log[ p(X|\lambda)p(S|\phi) ] p(X,S|Y, \phi',\lambda') dXdS$$

As noted by Xi'an the E step then considers

$$ Q(\lambda, \phi|\lambda',\phi') = E_{X,S|Y,\lambda',\phi'}[l(\lambda|X)] +E_{X,S|Y,\lambda',\phi'}[l(\phi|S)] $$

which could be maximized in turn in an expectation conditional maximization (ECM) algorithm. However, for the problem I am working on the joint predictive distribution $p(X,S|Y, \phi,\lambda)$ is not known. We can use the truncated distributions to write

$$ p(X|S,Y, \lambda) p(S|Y,\phi) = p(S|X,Y, \phi) p(X|Y,\lambda)$$

However the marginals $p(X|Y,\lambda),p(S|Y,\phi)$ are then still unknown for my problem. As Xi'an noted the log-likelihood simplifies to

$$ Q(\lambda, \phi|\lambda',\phi') = E_{X|Y,\lambda'}[l(\lambda|X)] +E_{S|Y,\phi'}[l(\phi|S)] $$

involving the same unknown densities.

Is there still a way to use a type of EM algorithm to solve this estimation problem?

If, for example, I would be allowed to augment $S$ and $X$ at some point (i.e. hold them fix at a given EM-step) it seems that $Q$ would only depend on the truncated distributions which can be calculated. However I am not sure if such holding fix the latent variates is allowed in EM.

In its general form the E-step of the EM algorithm finds the expectation

$$ Q(\theta|\theta') =\int \log[ p(Y,Z | \theta)] p(Z|Y,\theta') d Z$$

where $Y$ the data, $Z$ the latent variables, $\theta'$ the current parameters, and $l(\theta|Y,Z) = p(Y,Z|\theta)$ the complete data likelihood.

My general question is: does EM require us to know the joint conditional (predictive) distribution of the latent variables, $p(Z|Y,\theta)$?

Context. This question is phrased with more context as follows. Suppose $Z=[X,S]$ has two random variables parameterised by distinct vectors $\lambda$ and $\phi$ respectively, i.e. we know (can evaluate and sample from) $p(X,S|\lambda,\phi)=p(X|\lambda)p(S|\phi)$. For my problem, data $Y$ are a deterministic function $Y=f(X,S)$.

Variables $X,S$ are marginally independent; however they are not independent conditional on $Y$. We also know the complete data likelihood, it is

$$p(Y,X,S|\lambda, \phi) = p(X,S|\lambda, \phi) = p(X|\lambda)p(S|\phi).$$

Also we know the full conditionals $p(X|S,Y,\lambda)$ and $p(S|X,Y,\phi)$. They denote truncated distributions which can be derived. If we fill this in we find:

$$ \int \log[ p(X|\lambda)p(S|\phi) ] p(X,S|Y, \phi',\lambda') dXdS$$

As noted by Xi'an the E step then considers

$$ Q(\lambda, \phi|\lambda',\phi') = E_{X,S|Y,\lambda',\phi'}[l(\lambda|X)] +E_{X,S|Y,\lambda',\phi'}[l(\phi|S)] $$

which could be maximized in turn in an expectation conditional maximization (ECM) algorithm. However, for the problem I am working on the joint predictive distribution $p(X,S|Y, \phi,\lambda)$ is not known. We can use the (known) truncated distributions to write

$$ p(X|S,Y, \lambda) p(S|Y,\phi) = p(S|X,Y, \phi) p(X|Y,\lambda)$$

However the marginals $p(X|Y,\lambda),p(S|Y,\phi)$ are then still unknown for my problem. As Xi'an noted the log-likelihood simplifies to

$$ Q(\lambda, \phi|\lambda',\phi') = E_{X|Y,\lambda'}[l(\lambda|X)] +E_{S|Y,\phi'}[l(\phi|S)] $$

involving the same unknown densities.

Is there still a way to use a type of EM algorithm to solve this estimation problem?

If, for example, I would be allowed to augment $S$ and $X$ at some point (i.e. hold them fix at a given EM-step) it seems that $Q$ would only depend on the truncated distributions which can be calculated. However I am not sure if such holding fix the latent variates is allowed in EM.