Type checking the Bayes formula

Question

In the Bayes formula, $$ p(\theta|D) = \frac{ p(D|\theta) p(\theta) }{ p(D) } $$

• which of $D, \theta$ should we regard as given and which as variables,

• which of the factors $p(\theta|D), p(D|\theta), p(\theta), p(D)$ are valid probabilities

• which of $D, \theta$ can refer to more than one data point?

This is not for a class.

To explain the question:

$P(D|\theta)$ is the likelihood, which is described as proportional to a probability, but not itself a probability. I believe it is true that for fixed $\theta$, $\int p(D|\theta) d D = 1$, seems like this must be true. But with $D$ fixed and $\theta$ variable, the integral $\int p(D|\theta) d\theta$ is not necessarily one.

I assume that $p(\theta|D)$ is a probability, otherwise the normalizing constant in the denominator woudl not be necessary. I assume that $\int p(\theta|D) d\theta = 1$, but integrating the posterior with respect to $D, \int p(\theta|D) d D$, is not necessarily onee. Is that true?

Often $D$ is written in font $\mathcal{D}$ (look closely), which I believe is a hint that $D$ can describe one data point or multiple data points or all the data. For example, the likelihood of the data is often assumed to factor across individuald data points. However I believe $\theta$ is a single thing (though possibly a vector) regardless of the data.

Answer 1

• which of $D, \theta$ should we regard as given and which as variables,

Both are random variables, otherwise they wouldn't have probability distributions. Yet, it may be the case that some of the parameters are given, e.g. you estimate mean of normal distribution given the data, prior and known variance, so you have only two, not three, random variables to consider ($D$ and $\mu$, but not $\sigma^2$).

• which of the factors $p(\theta|D), p(D|\theta), p(\theta), p(D)$ are valid probabilities

All of them are valid probabilities, or probability densities. The likelihood $p(D|\theta)$ in Bayesian setting is conditioned on random variable (not fixed point, as in maximum likelihood), so it is a valid conditional distribution.

• which of $D, \theta$ can refer to more than one data point?

$D, \theta$ are random variables, not the observed points. You apply the Bayes theorem to two random variables and their distributions.

This may be more clear, it I give you an example, e.g. the (obligatory) coin example. Say that you've flipped a coin $n$ times and observed $k$ heads, given this data, you want to estimate the probability of throwing heads. This translates to the following model

$$ k|\pi \sim \mathcal{Bin}(n, \pi) \\ \pi \sim \mathcal{Beta}(\alpha,\beta) $$

where $k$ follows the binomial distribution (the likelihood) and we use beta distribution as a prior for $\pi$. In terms of Bayes theorem, this is

$$ p(\pi|k) \propto p(k|\pi) \; p(\pi) $$

where all the three components are random variables and all have distributions. In this case $n$ is fixed, so different authors would use different notational conventions and either include $n$ in the Bayes theorem formula, or not (I didn't). There are also two hyperparameters $\alpha,\beta$ that are fixed (assumed a priori).

Notice that this is a simple case and most of the textbooks, blogs, or tutorials, will describe such simple examples, while most of the real-life examples would get much more complicated (yet the same general principles apply). Also notice that Bayesian updating lets us process data all at once, or one point at a time, and both approaches would lead to same results. Many examples would describe simple case of updating given single, so instead of describing a model for some number of i.i.d. random variables ("you have ten samples of..."), it would be described in terms of single variable for simplicity.