# Can you write a probability based on the relative entropy?

Suppose we have a graphical model $X\rightarrow \Theta \rightarrow D$ where all the distributions are Gaussian Mixture Models. Suppose further that the distribution of $X$ has more components than the distribution of $\Theta$, so information is lost from $X$ to $D$.

In this case I observe $D$ and I want to find the marginal distribution of $X$ given $D$. I believe the way to do this is with hierarchical bayesian inference:

First Bayes rule: $$p(X|D) = \frac{p(D|X)p(X)}{p(D)}$$ But here the likelihood function $p(D|X)$ should be considered the marginal likelihood over all values of the latent variable $\Theta$: $$p(D|X) = \int_\Theta p(D|\Theta,X)p(\Theta|X)d\Theta$$

The first term in that integral should be pretty easy to compute, since $\Theta$ contains the parameters of a GMM: $$p(D|X,\Theta) = p(D|\Theta) = \prod_i \sum_k \pi_{ik}N(d_i|\mu_k,\Sigma_k)$$ What thing I can't figure out is how to compute $p(\Theta|X)$. Neither of these are observed, so we are talking about the probability of one distribution given another one. I understand that the KL divergence is one way to consider the difference between two PDFs. Can we use $D_{KL}$ as a probability?

$$p(\Theta|X) = \exp \left[-D_{KL}(\Theta||X)\right]$$ where $D_{KL}$ is the K-L divergence. However, this last part is purely based on the intuition that $\Theta$ should have a PDF that looks like that of $X$.

• You are confused about a lot of things. $p(\Theta|X)$ is just a shortcut for $p(\Theta=\theta|X=x)$. It's not the proba of one distribution given another one. It's the completely conventional notion of the probability distribution a random variable conditioning on a parameter. There is a relationship betwen $D_{KL}$ and the log of a probability distribution but it's unrelated to hierarchical models. – Guillaume Dehaene Sep 28 '15 at 8:42
• @Guillaume Dehaene tell me about it :) What's especially confusing to me is if $\Theta$ are the parameters of a GMM, what does $p(\Theta=\theta|X=x)$ mean? Put it another way, how can I express the notion that the latent GMM $\Theta$ is dependent on $X$ - in a sense by grouping components of $X$? – cgreen Sep 28 '15 at 8:45
• Are you familiar enough with model comparison before you dive into hierarchical models ? Model comparison is a little bit like discrete hierarchical models so you can build some intuition there – Guillaume Dehaene Sep 28 '15 at 9:03
• Also, a simple example for your generative model $X \rightarrow \theta$ is $X \sim N(0,1)$, $\sigma_k = \Gamma(\dots)$ and finally $\mu_k|(X=x) = N(x,1)$. (do you see what I mean by that ?) X simply tells you that all components are somewhat clustered together. That's not a good model for fitting data, but it's a simple example for understanding the method, because you can marginalize X explictly – Guillaume Dehaene Sep 28 '15 at 9:06
• It's a weird problem, I admit. $X$ is my interpretable quantity - it is a set of Gaussian density functions with unknown positions, and $D$ is what comes out of a data processing black box. $\Theta$ are the latent variables representing the loss of information in the data processing. I can infer possible GMMs $\Theta$ from $D$ by clustering - so the question is, how likely is it I would get particular values for $\Theta$ given some positions of $X$? – cgreen Sep 28 '15 at 9:15