Do sampling methods (MCMC/SMC) work for combination of continuous and discrete random variables?

Question

Consider a distribution $P = \frac{1}{2}P_1 + \frac{1}{2}P_2$ where $P_1, P_2$ are probability measures on a measurable space $(\mathbb R, \mathcal B)$ such that

$P_1: A \to \int_A \mathcal N(x; 0, 1) d\mu(x)$ for all $A \in \mathcal B$, and

$P_2 = \frac{1}{3} \delta_0 + \frac{2}{3} \delta_1$, i.e. a discrete distribution which assigns $0$ and $1$ probabilities $\frac{1}{3}$ and $\frac{2}{3}$.

What kind of proposal is chosen in this case and do we evaluate densities (as given in recipes for MCMC/SMC) or something else when calculating acceptance ratios (e.g. for Metropolis-Hastings) or importance weights (e.g. for Importance sampling, SMC, etc.)? Can you please point me to the correct resources?

Answer 1

$P$ has a density against a (reference) measure made of the Lebesgue measure plus the counting measure on $\{0,1\}$. The later measure gives weights of $1$ to the atoms $0$ and $1$. This means that the density at atoms like 0 and 1 is equal to the weight against the counting measure and only the counting measure!, hence it is 1/6 and 2/6 for 0 and 1, respectively, in your example. (See this other entry for additional comments on mixed measures.)

Therefore, MCMC and in particular Metropolis-Hastings apply to this case. This means that the proposal must also be continuous against the measure made of the Lebesgue measure plus the counting measure, hence the proposal can have atoms at 0 and 1. For the chain to be converging it must have atoms at 0 and 1.

Here is an example of a (dumb) Metropolis-Hastings algorithm for your target

#target is N(0,1) with prob 1/2, mass at 0 with prob 1/6 and at 1 with prob 1/3
targ=function(x,isint){
  if (isint){ t=(x==0)/6+(x==1)/3
    }else{ t=.5*dnorm(x)}
  return(t)
  }

#Metropolis with random walk+U{0,1} proposal
prop=function(val,isint){
  isint[2]*.25+.5*(1-isint[2])*dnorm(val[2],mean=val[1],sd=.3)
  }

T=1e5
samp=rep(NaN,T)
sampint=(runif(T)<.5)
samp[1]=runif(1)
sampint[1]=FALSE
for (t in 2:T){
  if (sampint[t]){ 
     samp[t]=as.integer(sample(c(0,1),1))
    }else{ 
      samp[t]=samp[t-1]+rnorm(1,0,.3)
      }
  metro=targ(samp[t],sampint[t])*prop(samp[t:(t-1)],sampint[t:(t-1)])/
        targ(samp[t-1],sampint[t-1])/prop(samp[(t-1):t],sampint[(t-1):t])
  if (runif(1)>metro){ samp[t]=samp[t-1];sampint[t]=sampint[t-1]}
  }

This gives the proper fit, as shown by the following histogram:

and the (right) proportion of zeros and ones:

> sum(sampint)/T
[1] 0.49676
> sum(samp==0)/sum(samp==1)
[1] 0.5027832