# How to do inference over two steps in a graphical model simultaneously?

I have observed data $D$ about a physical object described by $M$. I would like to determine the posterior distribution of $M$ given $D$, or $p(M|D)$. Now I can't infer this directly because unknown correlations in the observations. So I use parameters $\theta$ to smooth out the data $D$, reducing its effective "weight."

So now the problem to determining the posterior probabillity $p(M,\theta|D)$. Thus I need to optimize $M$ at the same time as $\theta$. I think the way to express this problem now is with the graphical model $M \rightarrow \theta \rightarrow D$. It seems that there are two linked inference problems:

1. inferring $\theta$ from $D$:

$$p(\theta|D) = \frac{p(D|\theta)p(\theta)}{p(D)}$$

2. inferring $M$ from $\theta$:

$$p(M|\theta) = \frac{p(\theta|M)p(M)}{p(\theta)}$$

How do I do both these inferences at the same time? Is there a simpler way to write an equation for the whole thing? It's really confusing because in (1), $\theta$ is the "model" but in (2) $\theta$ is the "data." Note that I have priors for both $M$ (keeping it physically reasonable) and $\theta$ (reducing its complexity).

Edit: My first attempt to solve this is with the "hierarchical Bayes" method: \begin{align} p(M,\theta|D) &\propto p(D|M,\theta)p(M,\theta) \\ & = p(D|M,\theta)p(\theta|M)p(M) \\ & = p(D|\theta)p(\theta|M)p(M) \end{align} Where the last step uses $D\perp M|\theta$ from the graphical model. While this is appealing, there's one thing missing: there should be a place for $p(\theta)$ as well as $p(M)$. That's because I can say something about the uncertainty parameters $\theta$ independent of everything - they should be as simple as possible. Maybe I need a different way to formulate the problem? Perhaps a different graphical model? Thanks for your help!

• What distributions are known exactly other than $p(M)$ and $p(\theta)$? – rwolst Mar 4 '15 at 12:36
• We know also $p(D|\Theta)$ and $p(\Theta|M)$. Although I'm not totally sure the difference between $p(\Theta|M)$ and $p(M|\Theta)$ in this case. – simurg Mar 4 '15 at 13:00
• I don't see the issue with using the hierarchical Bayes method here. The marginal distribution of the $\theta$ in this case is unnecessary. – rwolst Mar 4 '15 at 13:09