Understanding a derivation in the cross-validation literature The robust cross-validation method by Silva and Zanella (2022) looks like a promising piece of a solution to a model averaging problem I am working on. I am trying to understand the method in the manuscript, and I am having trouble deriving the proportionality relation in Equation 8. Is there something I am missing?
Using the notation in the paper, I would like to show:
$$
Z ^ {-1} \sum_{j =1}^n p(\theta)p(y_{-j} | \theta) \propto p(\theta | y) \left ( \sum_{j =1}^np(y_j | \theta) \right )^ {-1}
$$
Here, $\theta$ is a vector of model parameters, and $y = (y_1, \ldots, y_n)$ is a set of $n$ conditionally independent data observations. (Conditional on $\theta$, $y_i \perp\!\!\!\!\perp y_j$ for $i \ne j$, $1 \le i \le n$, $1 \le j \le n$.) $y_{-j}$ is the subset of $y$ with $y_j$ removed.
I begin with the definition of $Z$:
$$
Z = \sum_{j = 1}^n p(y_j | \theta) ^ {-1}
$$
and a known proportionality relation from importance sampling stated between Equations 4 and 5 of the manuscript:
$$
\frac{1}{p(y_i | \theta)} \propto \frac{p(\theta | y_{-i})}{p(\theta | y)}
$$
If I start from the left-hand side of the Equation 8, I am having trouble getting to the right-hand side.
$$
\displaylines{
Z ^ {-1} \sum_{j =1}^n p(\theta)p(y_{-j} | \theta) = Z ^ {-1} \sum_{j =1}^n p(y_{-j})p(\theta | y_{-j}) \qquad \text{(Bayes rule)} \\\ \propto Z ^ {-1} \sum_{j =1}^n p(y_{-j}) \frac{p(\theta | y)}{p(\theta | y_j)} \qquad \text{(Importance weight proportionality relation)} \\\ = Z ^ {-1} p(\theta | y) \sum_{j =1}^n p(y_{-j}) p(\theta | y_j)^{-1} \qquad \text{(Factor out the posterior)} \\\ = p(\theta | y) \sum_{j =1}^n \left ( \frac{p(y_{-j})}{\sum_{k = 1}^n p(y_{-k})} \right ) p(\theta | y_j)^{-1} \qquad \text{(Definition of Z)}
}
$$
Likewise in the reverse direction.
$$
\displaylines{
p(\theta | y) \left ( \sum_{j = 1}^n p(y_j | \theta)^{-1} \right ) \propto p(\theta | y) \left ( \sum_{j = 1}^n \frac{p(\theta | y_{-j})}{p(\theta | y)} \right ) \qquad \text{(Importance weight proportionality relation)} \\\ = \sum_{j = 1}^n p(\theta | y_{-j}) \qquad \text{(Cancel out the posterior)} \\\ = \sum_{j = 1}^n \frac{p(y_{-j} | \theta) p(\theta)}{p(y_{-j})} \qquad \text{(Bayes rule)} \\\ = Z ^{-1} \sum_{j = 1}^n Z \frac{p(y_{-j} | \theta) p(\theta)}{p(y_{-j})} \qquad \text{(Multiply and divide by Z)} \\\ = Z ^{-1} \sum_{j = 1}^n \left ( \frac{\sum_{k = 1}^n p(y_{-k})}{p(y_{-j})} \right ) p(y_{-j} | \theta) p(\theta) \qquad \text{(Definition of Z)}
}
$$
 A: I am Luca Silva, co-author of the paper and I will be more than happy to help you in understanding that formula. Before proceeding we have to remember that we are dealing with a distribution on $\theta$, hence all the quantities that will depend only on the observations can be incorporated in the proportionality constant. So the calculation to get from the left to the right side of the first formula are the following:
$$ Z^{-1}\sum_{j=1}^n p(y_{-j}|\theta)p(\theta) \propto \sum_{j=1}^n p(y_{-j}|\theta)p(\theta) = \sum_{j=1}^n \frac{p(y_j|\theta)}{p(y_j|\theta)}p(y_{-j}|\theta)p(\theta)=\sum_{j=1}^n \frac{p(y|\theta)p(\theta)}{p(y_j|\theta)}. $$
The first proportional can be done because we know that $Z$, no matter its form, is going to be independent of $\theta$. In the last equality I used the fact that $p(y_j|\theta)\cdot p(y_{-j}|\theta) = p(y|\theta)$ due to the assumed conditional independence of the likelihood. Now we are basically done, infact we can observe by Bayes theorem that
$$p(y|\theta)p(\theta) = p(\theta|y)p(y)\propto p(\theta|y).$$
The last proportional is given by the fact that snce $p(y)$ is a marginal probability it is independent of $\theta$, hence we can include it in the proportionality constant. Inserting this in the last mathematical expression we get
$$ \sum_{j=1}^n \frac{p(y|\theta)p(\theta)}{p(y_j|\theta)} \propto p(\theta|y) \sum_{j=1}^n p(y_j|\theta)^{-1}. $$
We where able to bring the posterior outside the sum as it is independent of the summation index $j$. That is it! Hope I was able to be clear, otherwise do not hesitate to tell me any perplexity left!
