Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have been reading a book that cites an example where a uniform distribution is the initial prior, and then a person scores 9/10 on a test. Then the resulting posterior becomes the prior distribution.

The book provides the following explanation for what happens, but despite consulting the Wikipedia page on integration I am struggling to understand exactly what is going on, and why this process is necessary.

enter image description here

share|improve this question

2 Answers 2

up vote 5 down vote accepted

Matthew's answer provides correct technical explanation. For intuitive understanding, you may think that to integrate over a distribution is just a fancy term for averaging over the distribution (or, taking the expectation over the distribution). In this answer, I consider the same model with discrete distributions (finite number of alternative $\theta$s), which may be easier to understand if one is not very familiar with integrating over continuous probability distributions. In this answer, for clarity I use $Pr$ to denote discrete probabilities and $p$ to denote probability densities - often $p$ is used to denote both like in the book excerpt ($k^{\mathrm{rep}}$ is a discrete random variable).

Discrete example

A discrete case may be easier to grasp, so let us consider a situation with only two types of answerers: those who guess at random (assume these are yes/no -questions), i.e., $\theta=0.5$ and those who are pretty good (say, $\theta=0.9$). Let our prior distribution for your rate-of-correct-answers be $Pr(\theta=0.5)=0.5$, $Pr(\theta=0.9)=0.5$. Let us now work out what is our posterior predictive probability that you get all five questions in the next test right. This is $k^{\mathrm{rep}} = 5\mid n^{\mathrm{rep}} = 5$. If you are a total-guesser, this probability would be \begin{equation} Pr(k^{\mathrm{rep}} = 5 \mid n^{\mathrm{rep}} = 5,\theta=0.5) = 0.5^5 \approx 0.03, \end{equation} while if you are pretty good at answering, the probability is \begin{equation} Pr(k^{\mathrm{rep}} = 5 \mid n^{\mathrm{rep}} = 5,\theta=0.9) = 0.9^5 \approx 0.59, \end{equation} but we don't know which of these classes of answerers you belong to! Therefore, our total probability for you getting 5/5 in the next test is obtained by weighting the previous probabilities by the probabilities of you belonging to either class. If we want to take into account our information about your success in the previous test ($n=10$ questions, $k=9$ correct answers), the correct probabilities to use for this weighting are the posterior probabilities given the previous data. I leave computing the posterior as an exercise, but the result is \begin{array}{l} Pr(\theta=0.5 \mid k=9,n=10) = \frac{0.5^9\,0.5}{0.5^9\,0.5 + 0.9^9\,0.1} \approx 0.025, \\ Pr(\theta = 0.9 \mid k=9,n=10) = \frac{0.9^9\,0.1}{0.5^9\,0.5 + 0.9^9\,0.1} \approx 0.975. \end{array} Our posterior predictive probability for getting all questions correctly is then \begin{array}{l} & Pr(k^{\mathrm{rep}}=5 \mid n^{\mathrm{rep}}=5,n=10,k=9) \\ = & Pr(\theta=0.5 \mid k=9,n=10)\times Pr(k^{\mathrm{rep}} = 5 \mid n^{\mathrm{rep}} = 5,\theta=0.5) \\+ &Pr(\theta=0.9 \mid k=9,n=10)\times Pr(k^{\mathrm{rep}} = 5 \mid n^{\mathrm{rep}} = 5,\theta=0.9) \\ \approx & 0.577, \end{array} which is pretty close to the predictive probability for the pretty-good class, but slightly adjusted downwards to take into account the small probability that your good score in the previous test was just luck and you actually are answering by guessing.

Moving to continuous case

The previous discrete case is a rather crude model - why on Earth would we find it likely that you have a correct-answer-rate of $\theta=0.9$, but impossible that you have $\theta=0.85$?? So, a better analysis will use a continuous prior distribution for $\theta$. In this case, all particular values have zero probability, and instead the prior and posterior distributions are described by densities $p(\theta)$, $p(\theta \mid n=10,k=9)$. The goal of obtaining the posterior predictive distribution is still the same, but now the weighted average prediction is obtained by integrating over the posterior distribution, as explained in Matthew's answer. If this concept is unfamiliar to you, you should read up on probability density functions.

Note also that in measure theory, averaging over both discrete and continuous distributions is formalized as integration, so the term 'integrate over posterior' may be used in the discrete case, too.

share|improve this answer

The goal here to get the posterior predictive distribution. Suppose we are given previous data $y$ for learning parameters $\theta$ by which we attain the posterior $\pi(\theta \mid y)$. But now we want to understand the distribution of $y^*$, a new (set of) observation(s), given the data we already have. That would be the distribution $\pi(y^* \mid y)$. How do we get this? We use the Law of Total Probability, which requires an integral: $$ \pi(y^* \mid y) = \int \pi(y^*, \theta \mid y) d\theta = \int \pi(y^* \mid \theta) \pi(\theta \mid y) d\theta. $$ $\pi(\theta \mid y)$ is the posterior distribution, so we are integrating over the posterior to get $\pi(y^* \mid y)$. We couldn't just use $\pi(y^* \mid \theta)$ for some point estimate of $\theta$ because that would not be accounting for the uncertainty we have about $\theta$ (and we could have a lot of uncertainty). Fortunately that uncertainty is entirely contained within the posterior $\pi(\theta \mid y)$.

For example, the Beta-Binomial distribution is the posterior predictive distribution for a binomial likelihood and beta prior obtained by the integral $$ \int \text{Binomial}(y^* \mid \theta) \text{Beta}(\theta \mid \alpha_{\text{updated}},\beta_{\text{updated}}) d\theta. $$

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.