Bayesian statistics, what is the predictive distribution?

Question

while doing some exercises I had a doubt about the last two points of this question:

Let $X_1, ... X_{20} |\theta \stackrel{iid}{\sim} Bern(\theta)$ and suppose $\theta \stackrel{}{\sim} Beta(2,2)$, namely:

$P[X_i = x | \theta] = \theta^x(1-\theta)^{1-x} \mathbb{1}_{0,1} (x)$ , and $g(\theta) = 6\theta(1-\theta)\mathbb{1}_{[0,1]}(\theta)$

(a) identify the posterior distribution of $\theta$, given $X_1=x_1, ... X_{20} = x_{20} $
(b) determine the predictive distribution of $X_{21}$, given the observed sample, namely $P[X_{21} = 1 | x_1,...x_{20}]$
(c) If the observed sample is such that $s_{20} = \sum_{i=1}^{20} x_i = 10$, determine $P[\theta < 0.5 | x_1,...x_{20}]$

I know that (a) is $f(\theta|x)$ but what is (b)? Is it just the marginal distribution of $X$?
And also, in (c) why do I need the observed sample?

Answer 1

The general form of the predictive distribution is $$\Pr(\tilde{x}|X)=\int_{\theta\in\Theta}f(\tilde{x}|\theta)\pi(\theta|X)\mathrm{d}\theta,\forall\tilde{x}\in\chi$$ where $\theta$ is the parameter and is an element of the parameter space $\Theta$, $\tilde{x}$ is the yet to be realized value, which is an element of $\chi$ which is the sample space, and $X=\{x_0\dots{x}_n\}$ is the observed sample.

In your case, your prior is $\beta(2,2)$. The posterior, when the sum is unknown, for your specific problem is $$\pi(\theta|x_1\dots{x}_{20})=\frac{\prod_{i-1}^{20}\theta^{x_i}(1-\theta)^{(1-x_i)}6\theta(1-\theta)}{\int_0^1\prod_{i-1}^{20}\theta^{x_i}(1-\theta)^{(1-x_i)}6\theta(1-\theta)\mathrm{d}\theta}.$$ Because of the choice of the prior, this radically simplifies to $$\frac{\theta^{\alpha+1}(1-\theta)^{n-\alpha+1}}{\mathrm{B}(\alpha+1,n-\alpha+1)},$$ where $$\alpha=\sum_{i=1}^{20}x_i$$, $n$ is the sample size, in this case 20, and $\mathrm{B}$ is the beta function.

Your posterior predictive distribution is a bit ugly to solve, but since it is one observation it becomes rather trivial.

$$\Pr(x_{21}=1|x_1\dots{x}_{20})=\frac{\alpha}{20}.$$

If $\alpha=10$, then $\Pr(\theta|\alpha=10)=.5$. Mechanically, the posterior in this case is $$\pi(\theta|x_1\dots{x}_{20})=16224936\theta^{11}(1-\theta)^{11}.$$

$$\int_0^{.5}16224936\theta^{11}(1-\theta)^{11}\mathrm{d}\theta=.5$$

Answer 2

For the answer to (b), in short, you are correct. You are trying to determine the probability of the next observation, $X_{21}$, a random variable, based on the actually observed values of the first twenty. Hence, the term predictive distribution. Note that the conditional statement is using lowercase letters, i.e. the sequence is written $x_{1}, x_{2}, ..., x_{20}$, which indicates that these are no longer random values but constants since they have actually been observed. Now, since you are given the assumption that the $X$ values are conditionally independent and identically distributed given the value for $\theta$, what does that mean about the information of any future value $X_{21}$ you gain from observing the actually realized values of the random trials $x_{1}, x_{2}, ..., x_{20}$? If the conditionally iid assumption holds not just for the first twenty, but an arbitrary finite sequence of observations, then an application of Bayes' rule, the result of (a), and some algebra should rigorously demonstrate your intuition.

For part (c), you are only given a partial restriction on the possible values of the outcomes - namely, that they sum to 10. However, again because of the conditional independence and identical distributions, this means that (inherited from the commutative property of multiplication of reals) that the sequence of observations is exchangeable. Thus, your distribution for $S_{20} = s_{20} = 10$ is a sum of Bernoulli random variables, which you should recognize as a Binomial distribution. Again, an application of Bayes' rule and the probability of the Binomial distribution taking on that value, perhaps with some algebraic manipulation of the result from (b) if you want to be exceedingly rigorous, should get you the result you require.

Just a quick comment on your notation, the indicator for the Bernoulli distribution is only non-zero when x is either 0 or 1, not any value in the closed interval $[0, 1]$, so it might be more explicit to use something like $1_{\{0, 1\}}$ even though it requires typing some escaped braces, especially since you also use an indicator variable for the continuous Beta distribution that actually uses the closed interval $[0, 1]$. But hey, abusing notation is fine as long as the intent is clear. Sorry if this answer was overly-long or ramblingly pedantic.