11
$\begingroup$

There are a lot of questions (like this) about some ambiguity with Bayesian formula in continuous case.

$$p(\theta | x) = \frac{p(x | \theta) \cdot p(\theta)}{p(x)}$$

Oftentimes, confusion arises from the fact that definition of conditional distribution $f(variable | parameter) $ is explained as $f$ being function of $variable$ given fixed $parameter$.

Alongside with that, there is an equivalence principle stating that likelihood can be written as: $$ L(\theta | x) = p(x | \theta)$$

So why not to use Bayes rule for distributions in the following form:

$$p(\theta | x) = \frac{L(\theta | x) \cdot p(\theta)}{p(x)}$$

to emphasize that we are dealing with functions of $\theta$ given observed data $x$, and that the respective term is likelihood (at least, starting with $L$)?

Is this a matter of tradition, or is there something more fundamental in this practice?

Improve this question
$\endgroup$
7
  • $\begingroup$ What is the meaning of $p(\cdot)$? I know this as a probability. But in the continuous case, I do not see what probability you are talking about. $\endgroup$
    – Sextus Empiricus
    Commented Jun 29, 2019 at 21:28
  • $\begingroup$ @MartijnWeterings, functions $p(\cdot)$ should be valid probability distributions in all cases except when it is "likelihood" of the form $p(x|\theta)$. Am I missing something? $\endgroup$
    – iot
    Commented Jun 29, 2019 at 21:33
  • $\begingroup$ What do you mean by probability distribution? Cumulative, density, etc.? $\endgroup$
    – Sextus Empiricus
    Commented Jun 29, 2019 at 21:33
  • 2
    $\begingroup$ It might help to step back and realize that there are no "variables" in Bayes theorem, at least as you use the term. There are data points and there are model parameters. In this sense, $P(model|data)P(data) = P(data,model)=P(data|model)P(model)$. You invoke a posterior-like creature $P(model|data)$ which you then call the likelihood. But it isn't. So I'm not sure where you are going with this. And in general $p(x|y) = p(y|x) \implies p(x)=p(y)$ which is nonsensical in the case where $x=data$ and $y=model.$ $x$ and $y$ don't even have the same support. $\endgroup$
    – Peter Leopold
    Commented Jun 29, 2019 at 22:03
  • 1
    $\begingroup$ Check stats.stackexchange.com/a/224299/35989 $\endgroup$
    – Tim
    Commented Jun 30, 2019 at 7:07

2 Answers 2

Reset to default
8
$\begingroup$

There are two basic results from probability that are at work in Bayes' theorem. One is a way of rewriting a joint probability density function:

$$p(x,\,y)=p(x\,|\,y)p(y).$$

The other is a formula for computing a conditional probability density function:

$$p(y\,|\,x)=\frac{p(x,\,y)}{p(x)}.$$

Bayes' theorem just stitches these two things together:

$$p(\theta\,|\,x)=\frac{p(x,\,\theta)}{p(x)}=\frac{p(x\,|\,\theta)p(\theta)}{p(x)}$$

So both the data $x$ and the parameters $\theta$ are random variables with joint pdf

$$p(x,\,\theta)=p(x\,|\,\theta)p(\theta),$$ and that's what shows up in the numerator in Bayes' theorem. So writing the likelihood as a conditional probability density instead of as a function $L$ of the parameters makes clear the basic probability at play.

That all said, you'll see people use either, like here or here.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ @iot In classical statistics, you might estimate the parameters by finding the $\theta$ that maximizes $p(x\,|\,\theta)$ as a function of $\theta$. So people will write $L(\theta)=p(x\,|\,\theta)$ and try to compute $\hat{\theta}_{MLE}={\arg\max}\,L(\theta)$. In this case, you don't care about $p(x\,|\,\theta)$'s "status" as a conditional pdf over $x$. You care about its status as a real-valued function of $\theta$ that you want to maximize with respect to $\theta$. So $L(\cdot)$-style notation is a holdover from that setting. $\endgroup$
    – jcz
    Commented Jun 29, 2019 at 22:23
13
$\begingroup$

The likelihood function is merely proportional to the sampling density, in the sense that you have $L_x(\theta) = k(x) \cdot p(x|\theta)$ for some constant $k(x) > 0$ (though you should note that the likelihood is a function of the parameter, not the data). If you want to use this in your expression for Bayes theorem then you need to include the same scaling constant in the denominator:

$$p(\theta|x) = \frac{L_x(\theta) \cdot p(\theta)}{k(x) \cdot p(x)} = \frac{L_x(\theta) \cdot p(\theta)}{\int L_x(\theta) \cdot p(\theta) \ d \theta} \propto L_x(\theta) \cdot p(\theta).$$

If you instead use the formula you have proposed, then you will end up with a kernel of the posterior density, but it may not integrate to one (and thus it is not generally a density).

Improve this answer
$\endgroup$
5
  • 2
    $\begingroup$ I like your answer, but in original formula $p(x|\theta)$ with $x$ being fixed (Bayesian context) does not also a valid probability distribution, and $p(x)$ also is a scaling factor not equal to 1. So, why do you think that $k$ is not unity in your explanation? $\endgroup$
    – garej
    Commented Jun 30, 2019 at 4:22
  • 1
    $\begingroup$ Quite often we formulate the likelihood function by removing multiplicative parts that do not depend on the parameter of interest. We do this in order to make analysis simpler, by avoiding the need to keep track of a constant of integration. For example, if $p(x|\theta) = \text{Bin}(x|n,\theta)$ then we would take $L_x(\theta) = \theta^x (1-\theta)^{n-x}$, removing the binomial coefficient in the binomial distribution. In this case we have $k = {n \choose x}$, which is not generally equal to one. $\endgroup$
    – Ben
    Commented Jun 30, 2019 at 5:11
  • 1
    $\begingroup$ so your point is that there is a convention that likelihood is usually free from unnecessary constants and so iot's version could be somewhat misleading for statisticians? $\endgroup$
    – garej
    Commented Jun 30, 2019 at 5:17
  • $\begingroup$ While that is indeed a conventional way to set the likelihood, the point here is that the likelihood function is generally defined only up to proportionality, so there is no guarantee that $k=1$ in the above working. $\endgroup$
    – Ben
    Commented Jun 30, 2019 at 6:46
  • $\begingroup$ @nbro: I have edited to clarify the sense in which I refer to proportionality, and to note that the likelihood is a function of the parameter, not the data. That is all that is needed in the present question. $\endgroup$
    – Ben
    Commented Nov 13, 2019 at 0:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.