Bayesian evidence with Sequential Monte Carlo and an unnormalized likelihood function: a contradiction?

Question

There is a contradiction in my understanding of Sequential Monte Carlo for estimating Bayesian evidence for model comparison:

Marginal likelihood (aka normalizing constant, aka Bayesian evidence) estimates are supposed to be directly "apples to apples" comparable between different simulations of different models with different parameters. This is the basis for Bayesian Model Selection (e.g. as described in E.T. Jaynes chapter 20.)

But,

SMC permits me to use an unnormalized likelihood function and, at least in the context of Bayesian parameter estimation using likelihood tempering, it seems like I can "spike" the marginal likelihood estimate to any value at all via my choice of unnormalized likelihood function.

This is a contradiction and it suggests a fundamental misunderstanding on my part. But what?

Concretely, suppose I choose an unnormalized likelihood function that just uniformly returns a big number:

L = 1000

Then I used a five-step likelihood tempering to derive a series of incremental likelihood functions:

L0 = L^0     = 1
L1 = L^0.25 ~= 6
L2 = L^0.5  ~= 31
L3 = L^0.75 ~= 178
L4 = L       = 1000

Since the likelihood is uniform it doesn't matter how many particles I have: they would all be equivalent anyway. So let's suppose there is just one.

SMC will do importance sampling between the likelihood functions and compute these weights:

W0 = L0               ~= 1
W1 = L1/L0 = 6/1      ~= 6
W2 = L2/L1 = 31/6     ~= 5.2
W3 = L3/L2 = 178/31   ~= 5.74
W4 = L4/L3 = 1000/178 ~= 5.61

and the marginal likelihood will just be the product of these incremental weights:

ML = 6 * 5.2 * 5.74 * 5.61 ~= 1000
                            = L

So there we have it: SMC estimates the normalizing constant / marginal likelihood to be the value of the unnormalized likelihood function, L, but that's just a number that I made up. It doesn't have any absolute/normalized meaning as a basis for comparison with other simulations from other models.

So where did I go wrong? How do I fix this approach so that the marginal likelihood value will be valid and practical for comparison between simulations of different models?

Answer 1

In Bayesian terminology, the marginal likelihood is the prior predictive density $$m(x)=\int_\Theta f(x|\theta)\pi(\theta)\,\text d\theta$$ where $f(\cdot|\cdot)$ is the sampling density, $\ell(\theta|x)=f(x|\theta)$ is the standard likelihood. This marginal likelihood integrates to one over the sample space $\mathcal X$: $$\int_\mathcal X m(x)\,\text dx=\int_\mathcal X \int_\Theta f(x|\theta)\pi(\theta)\,\text d\theta\,\text dx=1$$ It is also the normalising factor for the posterior density $$\pi(\theta|x) = \frac{f(x|\theta)\pi(\theta)}{m(x)}$$ and simulating from the posterior is feasible without deriving $m(x)$, provided $f(x|\theta)$ is known up to a constant (that is, not a constant indexed by $\theta$). This is also the case for tempered versions of the posterior density, e.g. $$\pi_i(\theta|x) = \frac{f(x|\theta)^{\tau_i}\pi(\theta)}{m_i(x)}\quad 0\le\tau_i\le1$$ SMC and other Monte Carlo methods (nested sampling, bridge sampling, harmonic mean, umbrella sampling, path sampling, &tc.) exploit a sample from $\pi_i(\theta|x)$ or from another distribution to approximate the normalising constant $m_i(x)$. When comparing several models through marginal likelihood ratios, the corresponding (standard) likelihoods must all be completely available (or up to the same multiplicative constant).