Bayesian inference - iterative updating with Bernoulli distribution (solved)

Suppose I pull samples from a Bernoulli distribution $$\mathcal{B}(\theta)$$

• I don't know the value of $$\theta$$, but in my case I know that $$\theta$$ can only have 11 discrete values, $$\theta \in \{0.0, 0.1, 0.2, \ldots, 0.9, 1.0\}$$

• I want to figure out the distribution $$P[\theta]$$ using Bayesian inference.

Because I know that $$\theta$$ can only have 11 possible values, $$P[\theta]$$ will be a discrete distribution over 11 values.

• So, I start with a non informative prior $$P_{0}[\theta]=\frac{1}{11}$$ for all $$\theta$$
• I iterate for each new sample, and I set $$P_{n+1}[\theta]=\frac{P[\text{data}\mid\theta]\space P_{n}[\theta]}{\sum P[\text{data}\mid\theta]\space P_{n}[\theta]}$$

So far so good... but when I use the Bernoulli distribution for $$P[\text{data}\mid\theta]$$, that is $$P[\text{sample}\mid\theta] = \theta^\text{sample} (1-\theta)^{1-\text{sample}}$$ the algorithm does not converge! 😱

Instead, the algorithm converges splendidly when I use the Binomial distribution for $$P[\text{data}\mid\theta]$$. That is $$P[\text{data}\mid\theta] = \theta^\text{successes}(1-\theta)^{n-\text{successes}}$$

Why do I need to use the Binomial distribution, to estimate the parameter of a Bernoulli distribution?

• I don't understand what exactly are you calculating and how, could you describe the exact procedure in greater details (e.g. show us your code)?
– Tim
Nov 15, 2018 at 21:27
• The sum of the values in the sample, which is your statistic, does not have a Bernoulli distribution once the sample size exceeds $1.$ One way to have this impressed forcibly on you is to substitute some plausible values in your probability formula: try $\theta=9/10$ and set sample (the sum of values) to $0.$ What does $\theta^\text{sample}(1-\theta)^{1-\text{sample}}$ equal in that case?
– whuber
Nov 15, 2018 at 22:19
• From your first graph, it appears that when you iterate, you are not updating your prior to be the posterior of the previous iteration. Remember that when you do iterative Bayesian updating, the prior for each new step is the posterior from the last iteration. Try making that change and see if it works.
– Ben
Nov 15, 2018 at 23:50
• @whuber thanks! but that's exactly my point. Why do I need the sum of the samples (Binomial) to estimate a Bernoulli. Isn't the prior itself able to carry all the information about the past Bernoulli samples? Nov 16, 2018 at 7:06

The fact that your first graph merely oscillates between two values suggests to me that you are resetting the prior each time you perform an iteration. So what you are seeing in the graph is a sequence of posteriors, each of which only take one data point into account. That is not the correct method for iterative Bayesian updating. Remember that when you do iterative Bayesian updating, the prior for each new iteration is the posterior from the last iteration. So your algorithm should be:

$$\pi_0(\theta) = \frac{1}{11} \quad \quad \quad \text{for all } \theta = \tfrac{0}{10}, \tfrac{1}{10}, ..., \tfrac{10}{10}.$$

For $$i=1,...,n$$ and $$x_i \in \{0,1\}$$, update your beliefs via the iteration:

$$\pi_i(\theta) = \frac{\theta^{x_i} (1-\theta)^{1-x_i} \pi_{i-1}(\theta)}{\sum_\theta \theta^{x_i} (1-\theta)^{1-x_i} \pi_{i-1} (\theta)}$$

Notice that in each iteration the prior $$\pi_{i-1}$$ is the posterior from the previous iteration. The mass function $$\pi_n$$ is the posterior after incorporating all the data.

• thanks @Ben! I use exactly the formula you suggested, but when I use $θ^{x_i}(1−θ)^{1−x_i}$ (Bernoulli) the posterior bounces back and forth, while when I use $θ^{successes_i}(1−θ)^{1−successes_i}$ (Binomial) the posterior converges. It seems the prior is unable to convey the memory of the past samples, in front a each new piece of evidence (0 or 1 in the Bernoulli case). By contrast in when the new piece of evidence is the sum of all past samples (Binomial case) the posterior converges. But here the memory is embedded in the data itself, the prior doesn't help much! Nov 16, 2018 at 6:58
• If the prior fails to convey the memory of the previous data, that is because you are not actually importing the previous posterior as the new prior. If you change your algorithm so that each iteration imports the previous posterior as its prior then the algorithm will "have memory" of the previous data and it should then look exactly like the binomial case.
– Ben
Nov 16, 2018 at 7:17

thanks @Ben for insisting that I check my update process. It was broken indeed, I was not updating $$P_{posterior}[\theta]$$ correctly.

So now I can proclaim that Bayesian inference with Bernoulli works perfectly and I post here the new chart to show it 🙂

• Good stuff thanks for posting an update - you might want to update your post title to avoid confusion in the future Nov 16, 2018 at 15:33
• @XavierBourretSicotte newbie question 😛 what would be a good title in this case? Shall I just append something "(SOLVED)" or change it to something else? Nov 16, 2018 at 19:11
• yeah I suppose, its not a big deal really, maybe something like Iterative Bayesian updating - Bernoulli convergence (solved) Nov 16, 2018 at 19:32
• Post a link to the code as well if you can - on your github or something, I would be interested in seeing it Nov 16, 2018 at 19:33
• @XavierBourretSicotte -- it's so simple I implemented it in a Excel sheet 😛 dropbox.com/s/nto6yak1w8ybjzg/bayesian-bernoulli.xlsx Nov 16, 2018 at 21:27