I'm working on a model with this basic structure:
The square nodes are data, and the round nodes are parameters and/or latent variables. The left plate represents the "training observations" we actually use for posterior calculations, and the right plate (with all the tildes) represents future "unseen observations".
It's basically an errors-in-covariates model: regress $Y$ on the unknown predictors $f$, for which we have proxy observations $X$. A relevant example of such a structure would be functional linear regression - in that case, $X_i$ is a vector of noisy observations from the unknown smooth function $f_i$, the parameter $\tau$ controls the smoothness of all $f$'s, and we have $Y_i = \int f_i(t) \beta(t) \mathrm{d}t$.
I'm struggling with the best way to measure predictive performance with this structure. The usual QOI would be the posterior predictive distribution
\begin{align} \mathbb{P}\left(\tilde{Y} \mid \tilde{X}, X, Y\right) = \int \mathbb{P}\left(\tilde{Y} \mid \tilde{f}, \beta\right) \mathbb{P}\left(\tilde{f}, \beta, \tau \mid \tilde{X}, X, Y\right) \mathrm{d}\tilde{f} \mathrm{d} \beta \mathrm{d} \tau. \end{align}
However, due to the shape of the DAG, in the PPD we'd have to account for posterior dependence between $\tau$ and $\tilde{X}$, so we'd no longer be measuring with respect to "completely unseen" future data. I think a more relevant quantity would be
\begin{align} \int \mathbb{P}\left(\tilde{Y} \mid \tilde{f}, \beta\right) \mathbb{P}\left(\tilde{f} \mid \tau,\tilde{X}\right) \mathbb{P}\left(\beta, \tau \mid X, Y\right) \mathrm{d}\tilde{f} \mathrm{d} \beta \mathrm{d} \tau. \end{align}
This way, the global parameters are only conditional on the training data and don't "see" the new $\tilde{X}$'s. There's just a couple problems I see with this idea:
I'm not sure if this quantity has any meaning in terms of the DAG structure.
Estimating this with Monte Carlo is challenging since I'm using HMC as implemented in Stan, and with the rest of this structure there's not really a way to get samples from the possibly-intractable $\mathbb{P}\left(\tilde{f} \mid \tau,\tilde{X}\right)$.
Regarding #2, in practice I'd be doing leave-one-out cross-validation, so in one fold the validation data in the right plate would consist of a single observation. Following this paper, I could run HMC with all the data and use importance sampling to estimate the LOO-CV score for the "unseen" observation, with weights given by
\begin{align} \frac{\mathbb{P}\left(\tilde{f} \mid \tau,\tilde{X}\right)\mathbb{P}\left(\tau, \beta \mid X, Y\right)}{\mathbb{P}\left(\tilde{f}, \beta, \tau \mid X, Y, \tilde{X}, \tilde{Y} \right)} \propto \frac{1}{\mathbb{P}\left(\tilde{Y} \mid \tilde{f}, \beta\right)\mathbb{P}\left(\tilde{X} \mid \tau\right)}. \end{align}
The only issue here is we'd have to do a second layer of Monte Carlo approximation $\mathbb{P}\left(\tilde{X} \mid \tau\right) \approx \frac{1}{T}\sum_{t=1}^T \mathbb{P}\left(\tilde{X} \mid \tilde{f}_t \right)$, with prior samples $\tilde{f}_t \sim \mathbb{P}\left(\tilde{f} \mid \tau\right)$. Is that a reasonable thing to do?
I realize this is a bit of a long post, but any help is greatly appreciated.