How to do prediction (evaluate marginal likelihood) in generative latent variable classifier?

Question

The dataset is $\{\boldsymbol x_t,y_t\}$ for $t=1,\dots,T$, where $y_t \in \{0,1\}$. Define a generative latent variable classifier whose plate diagram is shown above. For each data point, a local latent $z_t$ is associated, whose prior is parameterized by the hyperparameter $\eta$. The conditional probabilities $p(\boldsymbol x_t \mid z_t)$ and $p(y_t \mid z_t)$ are parameterized by fixed model parameters $\boldsymbol\theta$. The posterior $p(z_t \mid \boldsymbol x_t,y_t)$ is intractable due to an intractable evidence.

Fitting the model is straightforward, using variational EM. What have bugged me for a long time is how to make prediction on new data, i.e. finding the probability $p(y \mid \boldsymbol x,\boldsymbol\theta, \eta)$. Since $p(y \mid \boldsymbol x,\boldsymbol\theta,\alpha) \propto p(y,\boldsymbol x \mid \boldsymbol\theta,\eta)$, we need only to evaluate the latter form, i.e. the evidence. However, as mentioned above, the evidence is intractable:

$$ p(y,\boldsymbol x \mid \boldsymbol\theta,\eta) = \int p(z \mid \eta) p(y,\boldsymbol x \mid z,\boldsymbol\theta)\,\mathrm d z $$

where $z$ resides in some high dimensional space (Euclidean space with some constraints).

My attempts:

1. I tried naive Monte Carlo. I sample from $p(z \mid \eta)$, which is easy, and estimate the average of $p(y,\boldsymbol x \mid z,\boldsymbol\theta)$. This fails of course.
2. I tried importance sampling. The importance distribution is set to $q(z)$, where $q(z) \approx p(z \mid y,\boldsymbol x,\boldsymbol\theta,\eta)$ is found by variational inference (e.g., mean-field approximation). If $q(z)$ perfectly matches the posterior, there should be zero variance. Sampling from $q(z)$ is easy. Then I find the average $p(z \mid \eta)/q(z) p(y,\boldsymbol x \mid z,\boldsymbol\theta)$. Surprisingly, it seems that sampling for 100k times still does not yield good estimation for the evidence, since I cannot effectively discriminate between $\log p(y=0,\boldsymbol x \mid \boldsymbol\theta,\eta)$ and $\log p(y=1,\boldsymbol x \mid \boldsymbol\theta,\eta)$.
3. I know Metropolis-Hastings algorithm, which could be easier to work with in high dimension (Murphy, 2012). But I can't figure out how to apply it to my problem, since sampling from $p(z \mid \eta)$ is already an easy problem.
4. I make prediction directly using the Evidence Lower Bound (ELBO), which fails because it's too biased and is not discriminative enough.

I also find this problem kind of related to marginal likelihood estimation in VAE, if dropping the $y_t$ variable. In (Kingma & Welling, 2014), the authors claim that the marginal likelihood can be found only in low-dimensional setting.

My question: Is there a general guideline on how to make prediction with such generative latent variable classifier? I feel this should be an easy problem... or maybe I'm not on the right track -- should try something other than finding the marginal likelihood to make prediction?

Thank you so much for any help and suggestions!

---

EDIT:

I'd like to draw connection with latent factor regression (a.k.a. supervised PCA) (Murphy, 2012; chapter 12.5.1) to make this problem a bit more concrete. In latent factor regression, the (conditional) probabilities are all Gaussians: $p(z_t) = \mathcal N(\mathbf 0, \mathbf I)$, $p(y_t \mid z_t) = \mathcal N(\mathbf w_y^\top z_t + \mu_y, \sigma_y^2)$, $p(\boldsymbol x_t \mid z_t) = \mathcal N(\mathbf W_x z_t + \boldsymbol\mu_x, \sigma_x^2 \mathbf I)$. To convert to the model in this question, simply replace the Gaussian in $p(y_t \mid z_t)$ with a Bernoulli, e.g. $p(y_t \mid z_t) = \mathrm{Ber}(\mathrm{sigmoid}(\mathbf w_y^\top z_t + \mu_y))$.

Answer 1

It turns out that the issue is really easy, in that if $\boldsymbol x$ is independent with $z$, or if $y$ is independent with $z$ (e.g. when $p(y \mid z) \propto 1$), we cannot discriminate $y$ given $\boldsymbol x$: $p(y \mid x) = p(y)$. These may happen due to choice of hyperparameters.

In terms of the methods to estimate the marginal likelihood, according to (Fourment et al., 2020), ELBO (my 4th attempt) and VBIS (my 2nd attempt) are both valid choices to evaluate marginal likelihood, albeit VBIS is more stable than ELBO.