What are "prior distribution" and "posterior distribution" in the case of Bayesian statistics?
Can you give layman's examples?
I understand prior and posterior probabilities. However, I don't know "distribution".
$\begingroup$
$\endgroup$
4
-
$\begingroup$ I think you have it switched: when we say priors, we mean prior probability distributions. Priors are always defined as distributions. $\endgroup$– FirebugCommented Apr 6, 2022 at 6:42
-
$\begingroup$ @Firebug you can write down Bayes theorem for events so your comment may be unclear without further clarification. $\endgroup$– TimCommented Apr 6, 2022 at 8:26
-
$\begingroup$ By "distribution" do you mean densities (PDFs)? If yes, then the question of deriving Bayes' formula for densities from probabilities is indeed not trivial (see Papoulis 2002. Probabilities Random Variables and Stochastic Processes. Chp4-4). But if you mean CDFs then Bayes' theorem for CDFs follows immediately from Bayes' theorem for probabilities since CDFs are probabilities (i.e. needs only replacing the event with the random variable e.g. $P(A)$ with $P(X\le x)$). $\endgroup$– PaulGCommented Apr 6, 2022 at 9:23
-
$\begingroup$ @Tim since it's about Bayesian statistics I thought it followed directly from that definition, and not for the general Bayes' Theorem, which is not exclusively Bayesian $\endgroup$– FirebugCommented Apr 6, 2022 at 10:29
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Prior probability vs distribution is the same distinction as between probabilities (in general) and probability distributions. Probability is a number between 0 and 1, the probability distribution is a function $f(x)$ that maps some values $x$ to corresponding probabilities. Probabilities are usually not very interesting because they refer to binary events probability that $A$ is true $P(A)$, while probability distributions generalize this concept e.g. probability that after rolling a $K$-sided dice you'd observe $4$ as a result, $P(X=4)$. This introduces another concept: random variables. If those things are not familiar to you, I recommend a probability theory course or a handbook.
Bayes theorem is
$$ \underbrace{p(\theta \mid X)}_\text{posterior} = \frac{\overbrace{p(X \mid \theta)}^\text{likelihood} \, \overbrace{p(\theta)}^\text{prior}}{\underbrace{p(X)}_\text{normalizing constant}} $$
Prior is the unconditional probability of $\theta$ that you know a priori, given the data in likelihood it is updated so that you end up with conditional posterior. Next, you could use this posterior as a prior with new data, this is called Bayesian updating.
See also Help me understand Bayesian prior and posterior distributions.
$\begingroup$
$\endgroup$
Suppose you wonder about the percentage of likely voters who would currently vote for Candadate X in an upcoming election. X seems to get more favorable press than his opponent, but there have not yet been any professionally conducted polls.
Because elections for this office often tend to be close, you might think a reasonable prior distribution for X's share $p$ of the vote would be $\mathsf{Beta}(35, 30).$
This prior distribution has mean $\mu = 35/65 = 7/13 = 0.5385,$ and about 95% of its probability in the interval $(0.42, 0.66),$ and density function $f(x) \propto p^{36}(1-p)^{31}.$ where the symbol $\propto$ (read "proportional to") indicates that we have written the 'kernel' of the prior distribution without the "norming" constant that makes it integrate to unity over $(0,1).$ (The normalizing constant in @Tim's answer does not need to be evaluated.)
qbeta(c(.025,.975), 35,30)
[1] 0.4174763 0.6572035
Admittedly, many other similar prior distributions might have been chosen, but Bayesian statistics allows 'personal' probabilities, and this prior distribution, which gives a slight edge to Candidate X, while coming nowhere near suggesting he is sure to win, seems about right to you.
Next, suppose a reliable poll of $n = 500$ randomly chosen voters shows $x = 335$ in favor of Candidate X. This binomial likelihood function $$g(x|p) = {500 \choose 335}p^{335}(1 - p)^{165} \propto p^{335}(1 - p)^{165}.$$
[In frequentist statistics, the maximum likelihood estimate of $p$ based only on the poll would be $\hat p = 335/500 = 0.67.]$
According to Bayes' Theorem, the posterior distribution of $p$ is found by multiplying the prior distribution by the likelihood function $$h(p|x)\, =\, f(p)\times g(x|p)\, \propto\, p^{36}(1-p)^{31} \times p^{335}(1 - p)^{165}\\ \propto\, p^{391}(1-p)^{196},$$
where we recognize the final member above as the kernel of the distribution $\mathsf{Beta}(390, 195).$ Thus the posterior mean is $390/(290+196) \approx 0.802$ and a 95% posterior probability interval (credible interval) is $(0.608, 0.688).$ So, at this stage of the campaign Candidate X has a very comfortable lead.
qbeta(c(.025,.975),360,195)
[1] 0.6084775 0.6878051
The relatively uninformative prior has less influence on the posterior distribution than does the poll of 500 potential voters. However, the Bayesian credible interval applies to the current election. If we believe the prior distribution was realistic and that the poll is carefully done and free of bias, we can believe the credible interval.
Notes: (1) The beta prior distribution and the binomial likelihood are of compatible algebraic form. One says that they are conjugate. In this example, conjugacy makes it easy to find the posterior distribution.
(2) An Agresti-Cooil 95% CI for $p$ based only on the likelihood would be $(0.628, 0.710).$ But many frequentists would not associate this CI directly with the current election, but with the long run behavior of similar elections.
p.est = 337/504
p.est + qnorm(c(.025,.975))*sqrt(p.est*(1-p.est)/504)
[1] 0.6275571 0.7097445
