Joint "density" of data and indicators in Bayesian mixture model

Question

I'm currently working through the chapter on finite mixture models in BDA3 and came across the following model setup (with the usual slight abuse of notation):

Let $\lambda=(\lambda_1,\dots,\lambda_H)$, $\sum_{h=1}^H\lambda_h = 1$ be the mixing proportions and let $f(y_i\mid\theta_h)$ be some parametric density.

Then the observations $y=(y_1,\dots,y_n)$ can be modeled as a continuous real-valued random variable with density

$$p(y\mid \theta,\lambda) = \prod_{i=1}^n\sum_{h=1}^H \lambda_h f(y_i\mid\theta_h).$$

Alternatively we can introduce latent class indicators $z_{ih}$

$$z_{ih} = \begin{cases}1 & \text{if i-th observation is drawn from h-th mixture component} \\ 0 &\text{otherwise}\end{cases} $$

with distribution

$$z_i = (z_{i1},\dots,z_{iH}) \mid \lambda \sim \text{Multinomial}(1; \lambda)$$

and corresponding probability mass function $p(z_i\mid\lambda)$ which lets us write

$$p(y\mid z, \theta) = \prod_{i=1}^n \prod_{h=1}^H (f(y_i\mid\theta_h))^{z_{ih}}.$$

So far things seem clear. But then the authors write:

The joint distribution of the observed data $y$ and the unobserved indicators $z$ conditional on the model parameters can be written $$p(y, z\mid\theta,\lambda) = p(z\mid\lambda)p(y\mid z, \theta) = \prod_{i=1}^n\prod_{h=1}^H(\lambda_hf(y_i\mid\theta_h))^{z_{ih}}$$ with exactly one of $z_{ih}$ equaling $1$ for each $i$.

From the perspective of probability theory we can not just multiply $p(y\mid z,\theta)$ and $p(z\mid\lambda)$, as the former is the pdf of a continous random variable and the latter is the pmf of a discrete random variable. On the other hand, we can multiply with priors for $\lambda$ and $\theta$, work out the full conditionals and get a working Gibbs sampler.

So, my question is: What is $p(y, z\mid\theta,\lambda)$ as defined above, if it is neither a pdf nor pmf?

Answer 1

So, one year and a thorough course on probability later this seems kind of obvious, but maybe the following sketch is still helpful for someone:

• Probability density functions and probability mass functions are the same kind of object. Both are a Radon-Nikodym derivative of a measure; in the case of the pdf with regard to ($n$-dimensional) Lebesgue-Measure, in the case of the pmf with regard to a counting measure.
• The general case of Bayes' theorem relates the prior distribution to the Markov kernel that is the posterior "distribution" by way of a Markov kernel for the conditional distribution of the observables given the parameters. This need not involve any density or likelihood functions at all.
• The densities for the prior distribution and for the conditional distribution of observables given parameters can very well be taken with regard to different dominating measures. In that case the joint distribution of observations and parameters has a density with regard to the product measure of the dominating measures.
• In my case this means $p(y, z\mid\theta,\lambda)$ is a density of the conditional distribution of $y$ and $z$ given $\theta$ and $\lambda$ with regard to the product of Lebesgue measure and a counting measure on $\{1,\dots,H\}$.