Bayesian Inference and Conditional Probabilities

Question

From Wikipedia, under "Formal description of Bayesian inference:"

• $\theta$, the parameter of the data point's distribution, i.e., $x \sim p(x|\theta)$
• $\alpha$, the hyperparameter of the parameter, i.e., $\theta \sim p(\theta|\alpha)$.

I have a really hard time understanding the $p(x|\theta)$ and $p(\theta|\alpha)$ as conditional disitributions, especially the latter. Let me assume these are actually conditional pdfs. Then aren't the conditional pdfs defined in terms of the joint and marginal pdfs? I.e. would we not have the following? $$ p(x|\theta) = \frac{p(x, \theta)}{p(\theta)}, \qquad p(\theta|\alpha) = \frac{p(\theta, \alpha)}{p(\alpha)}. $$ But, there is no mention of either joint pdfs, nor do I see a mention of a joint pdf between the data point and the parameter, $p(x, \theta)$. This part I might be okay with, since we can view both $x$ and $\theta$ as random variables and only specify the conditional pdf $p(x|\theta)$ and not need to assume a joint density...right?

Furthermore, as I understand it, the hyperparameters are not random variables, so $p(\theta|\alpha)$, $p(\theta, \alpha)$ and $p(\alpha)$ are not even defined.

Perhaps this is just bad notation on Wikipedia's part? Would better notation be $\theta \sim p(\theta;\alpha)$?

Answer 1

For me, the point about Bayesian methods is that the parameter $\theta$ is uncertain, and its uncertainty (or degrees of belief about it taking different possible values) can be treated using the axioms of probability. This means it is possible to update beliefs about the distribution of $\theta$ by using the observed data $\mathbf{x}$ by combining the prior distribution for $\theta$ with the likelihood of observing $\mathbf{X}=\mathbf{x}$ given $\theta$ to find a posterior distribution for $\theta$ given $\mathbf{X}=\mathbf{x}$.

Again, for me, the hyperparameter $\alpha$ represents the underlying assumptions about $\theta$ which, although they may or may not be true, do not change in the analysis. Writing $p(\theta \mid \alpha)$ rather than $p(\theta)$ for the prior distribution is a useful reminder of these assumptions.

I do not particularly like the Wikipedia "formal" description. In particular I would distinguish the density for $\mathbf{X}$ more from the prior and posterior distributions for $\theta$, perhaps by writing $f(\mathbf{x}\mid \theta, \alpha)$ for the conditional distributions of the observations (proportional to the likelihood for the parameter given the observations) rather than $p(\mathbf{x}\mid \theta)$, and $\pi_0(\theta \mid \alpha)$ for the prior rather than $p(\theta \mid \alpha)$. This then leads to the posterior distribution of $\theta$ given the observed data being

$$\pi(\theta \mid \mathbf{x},\alpha) = \dfrac{\displaystyle \pi_0(\theta \mid \alpha) f(\mathbf{x}\mid \theta, \alpha) } {\displaystyle \int_{\theta \in \Theta} \pi_0(\theta \mid \alpha) f(\mathbf{x}\mid \theta, \alpha)\, d\theta } $$ which you could rewrite in measure-theoretic terms if you actually wanted to be formal. It might also be helpful to avoid using $\theta$ as a free variable in the numerator and a bound variable in the denominator.

If it is helpful to you, you might conceive $\displaystyle \int_{\theta \in \Theta} \pi_0(\theta \mid \alpha) f(\mathbf{x}\mid \theta, \alpha)\, d\theta$ as in some sense $p( \mathbf{x}\mid \alpha)$ after integrating out $\theta$ (or summing it out if it takes discrete values). You might then think of the main expression as a manipulation of something like $p( \mathbf{x}\mid \alpha) \pi(\theta \mid \mathbf{x},\alpha) =p( \mathbf{x}, \theta \mid \alpha) = \pi_0(\theta \mid \alpha) f(\mathbf{x}\mid \theta, \alpha)$. Personally I do not find this particularly useful (and it can tempt you to follow it down the rabbit hole to largely meaningless distractions). Instead I prefer to concentrate on checking that $\displaystyle \int_{\theta \in \Theta} \pi(\theta \mid \mathbf{x},\alpha)\, d\theta =1$.

Answer 2

Just because $\alpha$ isn't random doesn't mean $p(\alpha)$ isn't defined. It's simply a Dirac peak (or the indicator function) at the chosen value.

Since $p(\alpha)$ is indeed well defined, knowledge of $p(\theta | \alpha)$ and $p(x | \theta)$ gives us the full, joint distribution

$$p(x, \theta, \alpha) = p(x|\theta)p(\theta|\alpha)p(\alpha)$$