11
$\begingroup$

Kruschke's Bayesian book says, regarding the use of a beta distribution for flipping a coin,

For example, if we have no prior knowledge other than the knowledge that the coin has a head side and a tail side, that’s tantamount to having previously observed one head and one tail, which corresponds to a = 1 and b = 1.

Why would no information be tantamount to having seen one head and one tail -- 0 heads and 0 tails seems more natural to me.

$\endgroup$
  • 8
    $\begingroup$ (+1) The quotation is misleading because it invites the reader to equate two very different senses of "observe." The sense used here is that of having inspected the coin itself--in effect, it means you understand the experimental setup. But the conclusion that this implies $a=b=1$ depends on re-interpreting "observe" in the different sense of having run the experiment twice during which one outcome was heads and the other tails. This kind of logical sleight-of-hand is an intellectual cop-out; it only makes Bayesian methods appear arbitrary and logically slippery, which is a pity. $\endgroup$ – whuber Oct 2 '16 at 22:11
  • $\begingroup$ The quotation is wrong: there is no justification for a prior of Beta(1, 1). $\endgroup$ – Neil G Oct 3 '16 at 22:21
  • $\begingroup$ One could as easily argue that it's a single observations' worth of information -- half a head / half a tail. $\endgroup$ – Glen_b Oct 7 '16 at 9:41
  • 4
    $\begingroup$ Please keep in mind the intended purpose of that passage in the book. It's supposed to be a simple intuitive justification for beginning applied users, obviously not a mathematical argument and definitely not a claim that beta(1,1) is the best or only vague prior. Elsewhere in the book I take pains to show that modest variations in vague priors make no substantive difference in the posterior when the there is a moderately large amount of data. (Except for Bayes factors, of course, which are highly sensitive to the prior!) In other writings I've discussed the Haldane prior. $\endgroup$ – John K. Kruschke Dec 14 '16 at 21:01
16
$\begingroup$

The quotation is a "logical sleight-of-hand" (great expression!), as noted by @whuber in comments to the OP. The only thing we can really say after seeing that the coin has an head and a tail, is that both the events "head" and "tail" are not impossible. Thus we could discard a discrete prior which puts all of the probability mass on "head" or on "tail". But this doesn't lead, by itself, to the uniform prior: the question is much more subtle. Let's first of all summarize a bit of background. We're considering the Beta-Binominal conjugate model for Bayesian inference of the probability $\theta$ of heads of a coin, given $n$ independent and identically distributed (conditionally on $\theta$) coin tosses. From the expression of $p(\theta|x)$ when we observe $x$ heads in $n$ tosses:

$$ p(\theta|x) = Beta(x+\alpha, n-x+\beta)$$

we can say that $\alpha$ and $\beta$ play the roles of a "prior number of heads" and "prior number of tails" (pseudotrials), and $\alpha+\beta$ can be interpreted as an effective sample size. We could also arrive at this interpretation using the well-known expression for the posterior mean as a weighted average of the prior mean $\frac{\alpha}{\alpha+\beta}$ and the sample mean $\frac{x}{n}$.

Looking at $p(\theta|x)$, we can make two considerations:

  1. since we have no prior knowledge about $\theta$ (maximum ignorance), we intuitively expect the effective sample size $\alpha+\beta$ to be "small". If it were large, then the prior would be incorporating quite a lot of knowledge. Another way of seeing this is noting that if $\alpha$ and $\beta$ are "small" with respect to $x$ and $n-x$, the posterior probability won't depend a lot on our prior, because $x+\alpha\approx x$ and $n-x+\beta\approx n-x$. We would expect that a prior which doesn't incorporate a lot of knowledge must quickly become irrelevant in light of some data.
  2. Also, since $\mu_{prior}=\frac{\alpha}{\alpha+\beta}$ is the prior mean, and we have no prior knowledge about the distribution of $\theta$, we would expect $\mu_{prior}=0.5$. This is an argument of symmetry - if we don't know any better, we wouldn't expect a priori that the distribution is skewed towards 0 or towards 1. The Beta distribution is

    $$f(\theta|\alpha,\beta)=\frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha) +\Gamma(\beta)}\theta^{\alpha-1}(1-\theta)^{\beta-1}$$

    This expression is only symmetric around $\theta=0.5$ if $\alpha=\beta$.

For these two reasons, whatever prior (belonging to the Beta family - remember, conjugate model!) we choose to use, we intuitively expect that $\alpha=\beta=c$ and $c$ is "small". We can see that all the three commonly used non-informative priors for the Beta-Binomial model share these traits, but other than that, they are quite different. And this is obvious: no prior knowledge, or "maximum ignorance", is not a scientific definition, so what kind of prior expresses "maximum ignorance", i.e., what's a non-informative prior, depends on what you actually mean as "maximum ignorance".

  1. we could choose a prior which says that all values for $\theta$ are equiprobable, since we don't know any better. Again, a symmetry argument. This corresponds to $\alpha=\beta=1$:

    $$f(\theta|1,1)=\frac{\Gamma(2)}{2\Gamma(1)}\theta^{0}(1-\theta)^{0}=1$$

    for $\theta\in[0,1]$, i.e., the uniform prior used by Kruschke. More formally, by writing out the expression for the differential entropy of the Beta distribution, you can see that it is maximized when $\alpha=\beta=1$. Now, entropy is often interpreted as a measure of "the amount of information" carried by a distribution: higher entropy corresponds to less information. Thus, you could use this maximum entropy principle to say that, inside the Beta family, the prior which contains less information (maximum ignorance) is this uniform prior.

  2. You could choose another point of view, the one used by the OP, and say that no information corresponds to having seen no heads and no tail, i.e.,

    $$\alpha=\beta=0 \Rightarrow \pi(\theta) \propto \theta^{-1}(1-\theta)^{-1}$$

    The prior we obtain this way is called the Haldane prior. The function $\theta^{-1}(1-\theta)^{-1}$ has a little problem - the integral over $I=[0, 1]$ is infinite, i.e., no matter what the normalizing constant, it cannot be transformed into a proper pdf. Actually, the Haldane prior is a proper pmf, which puts probability 0.5 on $\theta=0$, 0.5 on $\theta=1$ and 0 probability on all other values for $\theta$. However, let's not get carried away - for a continuous parameter $\theta$, priors which don't correspond to a proper pdf are called improper priors. Since, as noted before, all that matters for Bayesian inference is the posterior distribution, improper priors are admissible, as long as the posterior distribution is proper. In the case of the Haldane prior, we can prove that the posterior pdf is proper if our sample contains at least one success and one failure. Thus we can only use the Haldane prior when we observe at least one head and one tail.

    There's another sense in which the Haldane prior can be considered non-informative: the mean of the posterior distribution is now $\frac{\alpha + x}{\alpha + \beta + n}=\frac{x}{n}$, i.e., the sample frequency of heads, which is the frequentist MLE estimate of $\theta$ for the Binomial model of the coin flip problem. Also, the credible intervals for $\theta$ correspond to the Wald confidence intervals. Since frequentist methods don't specify a prior, one could say that the Haldane prior is noninformative, or corresponds to zero prior knowledge, because it leads to the "same" inference a frequentist would make.

  3. Finally, you could use a prior which doesn't depend on the parametrization of the problem, i.e., the Jeffreys prior, which for the Beta-Binomial model corresponds to

    $$\alpha=\beta=\frac{1}{2} \Rightarrow \pi(\theta) \propto \theta^{-\frac{1}{2}}(1-\theta)^{-\frac{1}{2}}$$

    thus with an effective sample size of 1. The Jeffreys prior has the advantage that it's invariant under reparametrization of the parameter space. For example, the uniform prior assigns equal probability to all values of $\theta$, the probability of the event "head". However, you could decide to parametrize this model in terms of log-odds $\lambda=log(\frac{\theta}{1-\theta})$ of event "head", instead than $\theta$. What's the prior which expresses "maximum ignorance" in terms of log-odds, i.e., which says that all possible log-odds for event "head" are equiprobable? It's the Haldane prior, as shown in this (slightly cryptic) answer. Instead, the Jeffreys is invariant under all changes of metric. Jeffreys stated that a prior which doesn't have this property, is in some way informative because it contains information on the metric you used to parametrize the problem. His prior doesn't.

To summarize, there's not just one unequivocal choice for a noninformative prior in the Beta-Binomial model. What you choose depends on what you mean as zero prior knowledge, and on the goals of your analysis.

$\endgroup$
0
$\begingroup$

It's clearly incorrect. Observing 1 heads and 1 tails means that $p(\theta=0)=0$ (it's impossible to have an all-heads coin) and $p(\theta=1)=0$ (it's impossible to have an all-tails coin). The uniform distribution isn't consistent with this. What is consistent is a Beta(2,2). From the Bayesian solution to the coin-flip problem with a Laplace (i.e. uniform) prior on the $\theta$, the posterior probability is $p(\theta)={\rm Beta}(h+1,(N-h)+1)$.

$\endgroup$
  • $\begingroup$ I have a hard time understanding your answer. $\endgroup$ – Michael Chernick Apr 5 '17 at 11:18
  • $\begingroup$ Your conclusion that "the uniform distribution isn't consistent with this" is incorrect. It confuses density (which is what must be meant by "$p$") with probability. The (continuous) uniform distribution assigns zero probability to any atomic event such as $\theta=0$ or $\theta=1$. $\endgroup$ – whuber Apr 5 '17 at 15:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.