How to choose what to integrate out or what to condition on for marginal distributions?

Question

I am trying to work out the Bayesian posteriors of $\theta$, $\tau$ and the $\varepsilon$ in the following model: $$y(t) = \phi(t,\tau)\theta+v(t),$$ where $\{v(t)\}$ is an iid sequence of random variables with Gaussian distribution with zero mean and $\varepsilon$ variance. I assume Gaussian priors for the $\theta$ and $\tau$ and an arbitrary discrete prior for $\tau$. Using the Bayes' Theorem, I can say that the joint posterior distribution of these variables is $$P(\theta, \tau, \varepsilon|O)\propto L(\theta, \tau, \varepsilon|O)P(\theta, \tau, \varepsilon)$$ where $O$ represents the observations and $P(\theta, \tau, \varepsilon) = f(\theta)P(\tau)f(\varepsilon)$.

To calculate the marginal posterior distribution of $\theta$, I can integrate out the $\tau$ and $\varepsilon$. The only reason that I do this is because this is how I have learnt to find the marginal distributions. However, I have seen people conditioning on the other variables to obtain the posterior distribution of $\theta$. The same goes for the posterior distribution of $\tau$. Do I condition it on $\theta$, or $\varepsilon$ or both or do I integrate them out? There seems to be a gap in my understanding here. Any help here would be appreciated. Thanks.

Answer 1

Since the joint posterior writes $$P(\theta, \tau, \varepsilon|O) = \dfrac{ L(\theta, \tau, \varepsilon|O)P(\theta, \tau, \varepsilon)}{\sum_\iota\iint L(\eta, \iota, \epsilon|O)P(\eta, \iota, \epsilon)\text d\eta\,\text d\epsilon}$$ the exact derivation of the marginal [of the joint] posterior of $\theta$ follows as $$P(\theta|O) = \dfrac{\sum_\iota\int L(\theta, \iota, \epsilon|O)P(\theta, \tau, \varepsilon)\,\text d\epsilon}{\sum_\iota\iint L(\eta, \iota, \epsilon|O)P(\eta, \iota, \epsilon)\text d\eta\,\text d\epsilon}\tag{1}$$ and does not involve conditional distributions when the integration and summation steps in (1) are feasible.

As pointed out by @mlofton in their comment, conditionals may occur when simulation is required. For instance,a simulation-based approximation to this marginal is the so-called Rao-Blackwell or Rao-Blackwellized estimator averaging full conditionals $$\hat{P}(\theta|O) = \frac{1}{I}\sum_{i=1}^I P(\theta|\tau_i,\varepsilon_i,O)$$ when the sequence $(\theta_i,\tau_i,\varepsilon_i)_{i=1,\ldots,I}$ is iid from the joint posterior (or converging to simulations from the joint posterior when using an MCMC algorithm).