Updating priors based on outcome of Bernoulli trials I am observing Bernoulli trials: independent events which either occur (with a fixed probability $p$) or do not occur (probability $1-p$). My subjective prior belief is that $p=p_0$ (for non-triviality assume $0<p_0<1$.) If I then observe $s$ successes and $f$ failures, how should I update my estimate of $p$? Is there any way to make this problem well-defined?
 A: In Bayesian statistics you model the parameters as random variables, with a prior probability distribution, and you update that distribution based on data. Your $p$ (which I will call $\theta$ from now on) isn't really a random variable: you said you're 100% sure that it's equal to a deterministic constant. Or, to be more precise, you're saying that your prior belief is that $\theta$ is equal to $\theta_0$ with probability 1. Now, if you put all the probability mass on a single point, the posterior will be equal to the prior, no matter what the likelihood is, because from Bayes' theorem, the posterior is zero where the prior is zero, and it must have a total mass of 1. Use the Bayes' theorem when $\theta$ is a discrete random variable, which can assume the values $\theta_0,\dots,\theta_N$ with probabilities $p_0,\dots,p_N$:
$$p(\theta_0|y)=\frac{p(y|\theta_0)p_0}{\sum_{i=0}^Np(y|\theta_i)p_i}$$
if $0=N$ and thus $p_0=1$, you have
$$p(\theta_0|y)=\frac{p(y|\theta_0)p_0}{p(y|\theta_0)p_0}=1=p_0$$
To get out of this, you need to change your prior. Two simple proposals:


*

*You could say that you're 90% sure that $\theta=\theta_0$, but there
could also be a 10% chance that $\theta=\theta_1$, which is another
parameter value which you consider less likely, but not completely
implausible. Given your likelihood, updating is immediate:
$$p(\theta_0|y)=\frac{0.9p(y|\theta_0)}{0.9p(y|\theta_0)+0.1p(y|\theta_1)}=
\frac{0.9\theta_0^s(1-\theta_0)^f}{0.9\theta_0^s(1-\theta_0)^f+0.1\theta_1^s(1-\theta_1)^f}$$
Of course, you're not limited to just one extra plausible value. You
could add as many as you want, as long as the sum of the probability
masses is 1. And if you had infinitely many plausible values? This
leads naturally to the next prior proposal.

*You could consider $\theta$ as a continuous random variable, instead than a discrete one, but since you're extremely confident that $\theta=\theta_0$, you could use a prior which puts most of the mass near $\theta=\theta_0$, but also some mass on all the other values from 0 to 1. For example, a Beta prior with $\beta$ very high, and $\alpha=\frac{\beta \theta_0}{1-\theta_0}$, will give you a prior mean $\frac{\alpha}{\alpha+\beta}=\theta_0$, and a prior sample size of $\alpha+\beta=\frac{\beta \theta_0}{1-\theta_0}+\beta=\frac{1}{1-\theta_0}\beta$. If $\theta_0=0.2$, and $\beta=200$, you get

which is quite concentrated around $\theta_0$, but it gives a non-zero probability also to different values. The advantage of the Beta prior is that the Beta-Binomial family is conjugate, and since you have a Binomial likelihood, the posterior is also a Beta with updated shape parameters $\alpha^*=\alpha+s$ and $\beta^*=\beta+f$. Thus updating is again immediate.
