Generalization performance in Bayesian errors-in-covariates model

Question

I'm working on a model with this basic structure:

The square nodes are data, and the round nodes are parameters and/or latent variables. The left plate represents the "training observations" we actually use for posterior calculations, and the right plate (with all the tildes) represents future "unseen observations".

It's basically an errors-in-covariates model: regress $Y$ on the unknown predictors $f$, for which we have proxy observations $X$. A relevant example of such a structure would be functional linear regression - in that case, $X_i$ is a vector of noisy observations from the unknown smooth function $f_i$, the parameter $\tau$ controls the smoothness of all $f$'s, and we have $Y_i = \int f_i(t) \beta(t) \mathrm{d}t$.

I'm struggling with the best way to measure predictive performance with this structure. The usual QOI would be the posterior predictive distribution

\begin{align} \mathbb{P}\left(\tilde{Y} \mid \tilde{X}, X, Y\right) = \int \mathbb{P}\left(\tilde{Y} \mid \tilde{f}, \beta\right) \mathbb{P}\left(\tilde{f}, \beta, \tau \mid \tilde{X}, X, Y\right) \mathrm{d}\tilde{f} \mathrm{d} \beta \mathrm{d} \tau. \end{align}

However, due to the shape of the DAG, in the PPD we'd have to account for posterior dependence between $\tau$ and $\tilde{X}$, so we'd no longer be measuring with respect to "completely unseen" future data. I think a more relevant quantity would be

\begin{align} \int \mathbb{P}\left(\tilde{Y} \mid \tilde{f}, \beta\right) \mathbb{P}\left(\tilde{f} \mid \tau,\tilde{X}\right) \mathbb{P}\left(\beta, \tau \mid X, Y\right) \mathrm{d}\tilde{f} \mathrm{d} \beta \mathrm{d} \tau. \end{align}

This way, the global parameters are only conditional on the training data and don't "see" the new $\tilde{X}$'s. There's just a couple problems I see with this idea:

1. I'm not sure if this quantity has any meaning in terms of the DAG structure.

2. Estimating this with Monte Carlo is challenging since I'm using HMC as implemented in Stan, and with the rest of this structure there's not really a way to get samples from the possibly-intractable $\mathbb{P}\left(\tilde{f} \mid \tau,\tilde{X}\right)$.

Regarding #2, in practice I'd be doing leave-one-out cross-validation, so in one fold the validation data in the right plate would consist of a single observation. Following this paper, I could run HMC with all the data and use importance sampling to estimate the LOO-CV score for the "unseen" observation, with weights given by

\begin{align} \frac{\mathbb{P}\left(\tilde{f} \mid \tau,\tilde{X}\right)\mathbb{P}\left(\tau, \beta \mid X, Y\right)}{\mathbb{P}\left(\tilde{f}, \beta, \tau \mid X, Y, \tilde{X}, \tilde{Y} \right)} \propto \frac{1}{\mathbb{P}\left(\tilde{Y} \mid \tilde{f}, \beta\right)\mathbb{P}\left(\tilde{X} \mid \tau\right)}. \end{align}

The only issue here is we'd have to do a second layer of Monte Carlo approximation $\mathbb{P}\left(\tilde{X} \mid \tau\right) \approx \frac{1}{T}\sum_{t=1}^T \mathbb{P}\left(\tilde{X} \mid \tilde{f}_t \right)$, with prior samples $\tilde{f}_t \sim \mathbb{P}\left(\tilde{f} \mid \tau\right)$. Is that a reasonable thing to do?

I realize this is a bit of a long post, but any help is greatly appreciated.

Answer 1

The way you have drawn the DAG, the outcomes are conditionally independent of the predictors. So, the relevant quantity is still the posterior predictive distribution, it is just that the predictors only affect it through the posterior distribution of the unknown function. So, conceptually the steps to repeat are

1. Draw $\tau$ and $\beta$ from the posterior distribution
2. For each $i \in \{1, \dots, \tilde{N}\}$ draw $\tilde{f_i}$ from the prior distribution conditional on the posterior realization of $\tau$ and $\beta$ in step (1)
3. For each $i \in \{1, \dots, \tilde{N}\}$ draw $\tilde{Y_i}$ from the distribution used for the likelihood but now conditional on the posterior realization of $\tilde{f_i}$.

But even if you needed to condition on the predictors in some other model, they could be drawn from a posterior predictive distribution as well.