# How can we incorporating uncertainty about our data into Bayesian inference?

I want to use a Bayesian approach to estimate the parameter $$\theta$$ of a binomial distribution $$\textsf{Binomial}(\theta,n)$$ with the number of Bernoulli experiments $$x_i \in \{0,1\}$$ being known and fixed to $$n$$.

In the textbook example, the binomial likelihood only takes the sum of positive vs. negative outcomes of $$x_i$$ into account: $$$$f(x|\theta) = \theta^{\sum x_i} (1-\theta)^{n-\sum x_i}$$$$

Together with a uniform prior $$f(\theta)=\textsf{Uniform}(0,1)$$, the posterior becomes the Beta distribution $$$$f(\theta|s) = \textsf{Beta}(\theta|s+1,n-s+1) .$$$$ , where $$s=\sum x_i$$. Unfortunately, $$x_i$$ is not directly observable. Instead, we are given $$\pi_i$$ as features to e.g. a neural network, which then provides a probabilistic mapping $$f(x_i|\pi_i)$$. This probability represents uncertainty about the true, latent value $$x_i$$.

How can we incorporate this uncertainty about the data into the estimate? E.g. if $$f(x_i|\pi_i)=0.51\ \forall\ 0\leq\ i, we would expect the MAP estimate of $$\theta$$ to be a lot further off from $$1$$ than if $$f(x_i|\pi_i)=0.91$$ throughout.

• Is the input directly observable? Jun 7 '21 at 9:13
• @AccidentalStatistician Adjusted the problem statement to clarify this: $x_i$ is not directly observable. Jun 7 '21 at 14:40

The distribution for $$\theta$$ conditional on $$s$$ (and $$n$$) is $$$$p(\theta|s) = \textsf{Beta}(\theta|s+1,n-s+1) .$$$$ The problem is that we don't observe $$s$$.
Instead we are given $$\pi = (\pi_1, \ldots, \pi_n)$$, where $$$$p(x_i|\pi_i) = \textsf{Bernoulli}(x_i|\pi_i)$$$$ and $$$$p(x|\pi) = \prod_{i=1}^n p(x_i|\pi_i) .$$$$ We can compute $$p(s|\pi)$$ from $$p(x|\pi)$$ and then $$$$p(\theta|\pi) = \sum_{s=0}^n p(\theta|s)\,p(s|\pi) ,$$$$ which is a mixture of beta distributions.
The problem now is that $$p(s|\pi)$$ is the Poisson-binomial distribution which is difficult to compute analytically unless $$n$$ is quite small. One solution is to use simulation to approximate the distribution. In particular, let $$$$p(\theta|\pi) \approx \frac{1}{R} \sum_{r=1}^R p(\theta|s^{(r)}) ,$$$$ where $$s^{(r)} \sim p(s|\pi)$$.
Let me provide more detail. Let $$$$s^{(r)} = \sum_{i=1}^n x_i^{(r)} ,$$$$ where $$$$x_i^{(r)} \sim p(x_i|\pi_i) .$$$$ Now define the approximate mixture weights: $$$$\hat w_j = \frac{1}{R}\sum_{r=1}^R 1(s^{(r)} = j) ,$$$$ where the indicator function is $$$$1(x) = \begin{cases} 1 & \text{x is true} \\ 0 & \text{x is false} \end{cases} .$$$$ Then $$$$p(\theta|\pi) \approx \sum_{j=0}^n \hat w_j\,p(\theta|s=j) .$$$$ You will need to experiment to see how large $$R$$ needs to be so that you get a sufficiently accurate answer. Depending on $$n$$ and $$\pi$$, you might need $$R = 10^5$$ or more.
• Thanks for the second approach, which makes a lot of sense. I just need some clarification on the approximation: do I assume correctly that you suggest to sample $R$-times $s$ from $p(s|\pi)$? If so, how do I calculate an individual sample for $s \in [0,n]$ with $n \approx 3000?$ Jun 7 '21 at 14:12