Quadratic mean convergence of a biased coin using conditional expectation

Question

I'm a master's degree student and after a lot of research and some days trying I still can't get the answer for a question proposed by my statistics professor. Consider a sample of coin tosses:

$$X_1, X_2, ..., X_n | P=p \sim \text{IID Bern}(p),$$

where the coin tosses are conditionally independent Bernoulli random variables with probability given by $P$ (a head on the coin is a one and a tail is a zero). Denote the sample mean of the coin tosses as:

$$S_n = \frac{1}{n} \sum_{i=1}^n X_i.$$

My professor claims that $\mathbb{E}(S_n) = \mathbb{E}(X_1) = \mathbb{E}(P)$, and says that it is possible to use conditional expectation to prove this. He also asks me to prove that $S_n$ converges to $P$ in quadratic mean (not just to its expected value). How can I prove this?

Answer 1

This is a classic case in Bayesian analysis, where you have an exchangeable sequence of observations that are conditionally independent given the underlying parameters, but they are not marginally independent when you do not condition on the parameters. You can find simple explanations of this phenomenon, discussing the Bernoulli model as an example, in O'Neill (2009). You can also find some more aspects of Bayesian analysis of this Bernoulli model in a related series of papers (O'Neill and Puza 2005, O'Neill 2012, and O'Neill 2015).

From the properties of the Bernoulli distribution, you have the conditional mean $\mathbb{E}(X_i|P) = P$. Using the law of iterated expectations, you have:

$$\begin{equation} \begin{aligned} \mathbb{E}(S_n) = \mathbb{E} \Big( \mathbb{E}(S_n|P) \Big) &= \mathbb{E} \Bigg( \mathbb{E} \Bigg( \frac{1}{n} \sum_{i=1}^n X_i \Bigg| P \Bigg) \Bigg) \\[6pt] &= \mathbb{E} \Bigg( \frac{1}{n} \sum_{i=1}^n \mathbb{E} (X_i | P ) \Bigg) \\[6pt] &= \mathbb{E} \Bigg( \frac{1}{n} \sum_{i=1}^n P \Bigg) \\[6pt] &= \mathbb{E} ( P ). \\[6pt] \end{aligned} \end{equation}$$

Similarly, you have:

$$\begin{equation} \begin{aligned} \mathbb{E}(X_i) = \mathbb{E} \Big( \mathbb{E}(X_i|P) \Big) &= \mathbb{E}(P). \\[6pt] \end{aligned} \end{equation}$$

This establishes the equivalence of the expected values asserted by your professor. For the latter problem, you will need to prove that $\mathbb{E}((S_n-P)^2) \rightarrow 0$ as $n \rightarrow \infty.$ I will not give you a solution here, but as a hint, I recommend that you use the law of iterated expectation to get an expression for the expectation in the statement. You should then be able to show that this expression converges to zero in the limit, which would establish convergence in quadratic mean.

Answer 2

I think your question is not quite clear. First, it's hard for me to see how coin flips can not be independent. Are you saying that somehow the coin flips depend on each other? What I think that you mean is that the coin flips are not iid (they are independent but not identically distributed). Here that would mean that the parameter P changes with each flip of the coin.

$$E[S_n] = E[E[S_n|P=p]] = E\left[E\left[\frac{1}{n} \sum_{i=1}^n X_i | P \right] \right] = E[P]$$

The first equality is the law of iterated expectations. The second is just substituting for $S_n$. The third is because $X_i \sim Bernoulli(P) \implies E[X_i|P=p] = p$.

For convergence in quadratic mean, you need to show that:

$0 = \lim_{n \rightarrow \infty} E[(S_n - P)^2]$

$=\lim_{n \rightarrow \infty} E[E[(S_n - P)^2|P]]$

$= \lim_{n \rightarrow \infty} E[Var(S_n | P)]$

$= \lim_{n \rightarrow \infty} E\left[Var\left(\frac{1}{n} \sum_{i=1}^n X_i | P) \right) \right]$

$= \lim_{n \rightarrow \infty} E\left[\frac{1}{n^2} \sum_{i=1}^n Var(X_i | P)\right]$

$= \lim_{n \rightarrow \infty} \frac{1}{n} E[P(1-P)]$

$= 0$