I am currently studying the textbook In All Likelihood -- Statistical Modelling and Inference Using Likelihood by Yudi Pawitan. Section Inverse probability: the Bayesians of chapter 1 says the following:

The first modern method to assimilate observed data for quantitative inductive reasoning was published (posthumously) in 1763 by Bayes with his Essay towards Solving a Problem in the Doctrine of Chances. He used an inverse probability, via the now-standard Bayes theorem, to estimate a binomial probability. The simplest form of the Bayes theorem for two events $A$ and $B$ is

$$P(A \vert B) = \dfrac{P(AB)}{P(B)} = \dfrac{P(B \vert A)P(A)}{P(B \vert A)P(A) + P(B \vert \overline{A})P(\overline{A})}. \tag{1.1}$$

Suppose the unknown binomial probability is $\theta$ and the observed number of successes in $n$ independent trials is $x$. Then, in modern notation, Bayes's solution is

$$f(\theta \vert x) = \dfrac{f(x, \theta)}{f(x)} = \dfrac{f(x \vert \theta) f(\theta)}{\int f(x \vert \theta) f(\theta) d \theta}, \tag{1.2}$$

where $f(\theta \vert x)$ is the conditional density of $\theta$ given $x$, $f(\theta)$ is the so-called prior density of $\theta$ and $f(x)$ is the marginal probability of $x$. (Note that we have used the symbol $f(\cdot)$ as a generic function, much like the way we use $P(\cdot)$ for probability. The named argument(s) of the function determines what the function is. Thus, $f(\theta, x)$ is the joint density of $\theta$ and $x$, $f(x \vert \theta)$ is the conditional density of $x$ given $\theta$, etc.)

Leaving aside the problem of specifying $f(\theta)$, Bayes had accomplished a giant step: he had put the problem of inductive inference (i.e. learning from data $x$) within the clean deductive steps of mathematics. Alas, 'the problem of specifying $f(\theta)$' a priori is an equally giant point of controversy up to the present day.

There is nothing controversial about the Bayes theorem (1.1), but (1.2) is a different matter. Both $A$ and $B$ in (1.1) are random events, while in the Bayesian use of (1.2) only $x$ needs to be a random outcome; in a typical binomial experiment $\theta$ is an unknown fixed parameter. Bayes was well aware of this problem, which he overcame by considering that $\theta$ was generated in an auxiliary physical experiment - throwing a ball on a level square table - such that $\theta$ is expected to be uniform in the interval $(0, 1)$. Specifically, in this case we have $f(\theta) = 1$ and

$$f(\theta \vert x) = \dfrac{\theta^x(1 - \theta)^{n - x}}{\int_0^1 u^x(1 - u)^{n - x} du} \tag{1.3}$$

Equation (1.3) is obviously an application of Bayes' theorem. And the terms in the numerator and denominator seem to be the binomial PMF.

  1. It's been a while since I dealt with conditional probabilities, but, if my understanding is correct, this means that the random variable $\theta$ in $f(\theta \vert x)$ is binomially distributed? Or does it mean that the entire conditional distribution $f(\theta \vert x)$ is binomially distributed? Sorry for the elementary question, but I'm trying to remember the correct way to think about this.

  2. Given what I said in 1. regarding the binomial PMF, why is $u$ used for the probability values in the integral in the denominator, whereas $\theta$ is used for the same probability value in the numerator?

I would greatly appreciate it if people would please take the time to clarify this.

  • $\begingroup$ I think , $f(\theta |x)$ is $\beta (x+1,n-x+1)$ distribution,Beta, not binomial. $\endgroup$ – Masoud Feb 18 '20 at 12:14
  • $\begingroup$ @masoud Hmm, I don't see how what's in (1.3) matches the beta PDF? en.wikipedia.org/wiki/… $\endgroup$ – The Pointer Feb 18 '20 at 12:16
  • $\begingroup$ note $\int_0^{1} u^{a-1} (1-u)^{b-1} du=\beta(a,b)=\frac{\Gamma(a) \Gamma(b)}{\Gamma(a+b)}$ $\endgroup$ – Masoud Feb 18 '20 at 12:20
  • $\begingroup$ dear @The Pointer, I see $\theta \in (0,1)$ so I think it can not be binomial. $\endgroup$ – Masoud Feb 18 '20 at 12:34
  • 1
    $\begingroup$ ,Yes. that is right $\endgroup$ – Masoud Feb 18 '20 at 12:47


$\int_0^{1} u^{a-1} (1-u)^{b-1} du=\beta(a,b)=\frac{\Gamma(a) \Gamma(b)}{\Gamma(a+b)}$

$$f(\theta |x)=\frac{\theta^{x} (1-\theta)^{n-x}}{\int_0^{1} u^{(x+1)-1} (1-u)^{(n-x+1)-1} du} =\frac{\theta^{(x+1)-1} (1-\theta)^{(n-x+1)-1}}{\beta(x+1,n-x+1)}$$ $$\theta \in (0,1)$$

For question 2)

In the part

$$\int_0^{1} u^{a-1} (1-u)^{b-1} du$$

It does not matter use $u$ or any symbol. On the other hand

$$\int_0^{1} u^{a-1} (1-u)^{b-1} du=\int_0^{1} t^{a-1} (1-t)^{b-1} dt$$

  • $\begingroup$ the prior of $\theta$ is beta, the posterior also is beta. just updated the distribution based on observation.It mean $f(\theta|x)$ is beta,(the posterior distribution that updated by observation $\endgroup$ – Masoud Feb 18 '20 at 12:43
  • $\begingroup$ See my other comment. Doesn't the author say that $\theta$ is uniformly distributed on the interval $(0, 1)$? $\endgroup$ – The Pointer Feb 18 '20 at 12:46
  • $\begingroup$ in this : $f(\theta)=1$ and note $\theta \in (0,1)$ $\endgroup$ – Masoud Feb 18 '20 at 12:48
  • $\begingroup$ Ok, thank you for the clarification. $\endgroup$ – The Pointer Feb 18 '20 at 12:49
  • $\begingroup$ you are well come $\endgroup$ – Masoud Feb 18 '20 at 12:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.