Bayes theorem: a function of D, or theta, or both?

Question

In the Bayes formula as written for machine learning applications, $$ p(\theta|D) = \frac{ p(D|\theta) p(\theta) }{ p(D) } $$ where $D$ is the data, $\theta$ are the model parameters.

Commonly $p(\theta)$ is labeled the prior, $p(D|\theta)$ is called the likelihood, and $p(D)$ is called the evidence (or marginal likelihood I think).

The question: I am bothered by calling $p(D|\theta)$ as a likelihood. I believe a likelihood is not a probability density, meaning that $\int p(D|\theta) d\theta$ does not integrate to one. Wheras I think $\int p(D|\theta) d\,D$ does integrate to one.

So it seems strange to include something that is not a probability in the formula. I think maybe my confusion is about whether the overall Bayes formula should be considered as "a function of" $D$, or $\theta$, or either or both?

Answer 1

In probability theory (as this has nothing specific to machine learning or statistics) Bayes' formula is based on the joint distribution of the pair of random variables $(D,\theta)$, $p(D,\theta)$, which can be expressed either as $$p(D,\theta)=p(D)\times p(\theta|D)$$ or as $$p(D,\theta)=p(\theta)\times p(D|\theta)$$ [with the confusion notation of using the same $p(\cdot)$ everywhere!] where

1. $p(D)$ denotes the marginal density of $D$ [integrates to one in $D$] also called marginal likelihood or evidence
2. $p(\theta)$ denotes the marginal density of $\theta$ [integrates to one in $\theta$] also called prior
3. $p(D|\theta)$ denotes the conditional density of $D$ given $\theta$ [integrates to one in $D$] also called likelihood and often denoted $\ell(\theta)$
4. $p(\theta|D)$ denotes the conditional density of $\theta$ given $D$ [integrates to one in $\theta$] also called posterior