3
$\begingroup$

In the Bayes formula as written for machine learning applications, $$ p(\theta|D) = \frac{ p(D|\theta) p(\theta) }{ p(D) } $$ where $D$ is the data, $\theta$ are the model parameters.

Commonly $p(\theta)$ is labeled the prior, $p(D|\theta)$ is called the likelihood, and $p(D)$ is called the evidence (or marginal likelihood I think).

The question: I am bothered by calling $p(D|\theta)$ as a likelihood. I believe a likelihood is not a probability density, meaning that $\int p(D|\theta) d\theta$ does not integrate to one. Wheras I think $\int p(D|\theta) d\,D$ does integrate to one.

So it seems strange to include something that is not a probability in the formula. I think maybe my confusion is about whether the overall Bayes formula should be considered as "a function of" $D$, or $\theta$, or either or both?

$\endgroup$
6
$\begingroup$

In probability theory (as this has nothing specific to machine learning or statistics) Bayes' formula is based on the joint distribution of the pair of random variables $(D,\theta)$, $p(D,\theta)$, which can be expressed either as $$p(D,\theta)=p(D)\times p(\theta|D)$$ or as $$p(D,\theta)=p(\theta)\times p(D|\theta)$$ [with the confusion notation of using the same $p(\cdot)$ everywhere!] where

  1. $p(D)$ denotes the marginal density of $D$ [integrates to one in $D$] also called marginal likelihood or evidence
  2. $p(\theta)$ denotes the marginal density of $\theta$ [integrates to one in $\theta$] also called prior
  3. $p(D|\theta)$ denotes the conditional density of $D$ given $\theta$ [integrates to one in $D$] also called likelihood and often denoted $\ell(\theta)$
  4. $p(\theta|D)$ denotes the conditional density of $\theta$ given $D$ [integrates to one in $\theta$] also called posterior
$\endgroup$
  • $\begingroup$ This seems intuitive. (Typo in #4, "DS"). However I am bothered by calling $p(D|\theta)$ a likelihood if it is viewed as an object where we integrate D, and so it seems to be a function of D rather than of $\theta$. I am sure this comment is stupid, but that is my feeling. $\endgroup$ – basicidea Nov 29 '18 at 18:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.