# Expectation and variance of bootstrap samples

## Background

The data at hand consists of $$n$$ iid random variables represented as $$X_j$$, where $$j \in \{1,\ldots,n\}$$. We know $$\forall i,\, \operatorname{E}\left(X_i\right) = \mu$$, and that $$\operatorname{Var}\left(X_i\right) = \sigma^2$$.

Suppose we generate $$B$$ bootstrap samples from this data, with the $$i$$th element of the $$b$$th bootstrap sample denoted by $$X_i^{∗b}$$.

## Question

What are $$\operatorname{E}\left(X_i^{∗b}\right)$$ and $$\operatorname{Var}\left(X_i^{∗b}\right)$$?

## (i) Showing that $$\operatorname{E}\left(X_i^{∗b}\right) = \mu$$
$$X_i^{∗b}$$ is "composed" of random variables. To simplify the process of finding $$\operatorname{E}\left(X_i^{∗b}\right)$$, start with a conditional expectation.
$$\begin{equation*} \operatorname{E}\left(X_i^{∗b}\,\vert\,X_1,\ldots,X_n\right) = \sum_{j = 1}^{n}\frac{X_j}{n} \end{equation*}$$ This expected value is a random variable itself. Therefore: $$\begin{equation*} \begin{split} \operatorname{E}\left(X_i^{∗b}\right) &= \operatorname{E}\left[\operatorname{E}\left(X_i^{∗b}\,\vert\,X_1,\ldots, X_n\right)\right]\\ &= \operatorname{E}\left(\sum_{j = 1}^{n}\frac{X_j}{n}\right)\\ & = \frac{1}{n}\sum_{j = 1}^{n}\operatorname{E}\left(X_j\right)\\ & = \frac{1}{n}\sum_{j = 1}^{n}\mu\\ & = \mu.\,\square \end{split} \end{equation*}$$
## (ii) Showing that $$\operatorname{Var}\left(X_i^{∗b}\right) = \sigma^2$$
Recall that: $$\begin{equation*} \begin{split} \operatorname{Var}\left(X_i^{∗b}\right) &= \operatorname{E}\left[\left(X_i^{∗b}\right)^2\right] - \left[\operatorname{E}\left(X_i^{∗b}\right)\right]^2\\ &= \operatorname{E}\left[\left(X_i^{∗b}\right)^2\right] - \mu^2. \end{split} \end{equation*}$$ Let us find $$\operatorname{E}\left[\left(X_i^{∗b}\right)^2\right]$$ in a manner similar to (i). We will need to also apply the law of the unconscious statistician'', which states that: $$\begin{equation*} \operatorname{E}[g(X)] = \sum _{x}g(x)f_{X}(x) \end{equation*}$$ for some function $$g$$ acting on a random variable $$X$$. $$\begin{equation*} \operatorname{E}\left[\left(X_i^{∗b}\right)^2\,\bigg|\,X_1,\ldots,X_n\right] = \sum_{j = 1}^{n}\frac{\left(X_j\right)^2}{n} \end{equation*}$$ $$\begin{equation*} \begin{split} \implies \operatorname{E}\left[\left(X_i^{∗b}\right)^2\right] &= \operatorname{E}\left\{\operatorname{E}\left[\left(X_i^{∗b}\right)^2\,\bigg|\, X_1,\ldots,X_n\right]\right\}\\ &= \operatorname{E}\left[\sum_{j = 1}^{n}\frac{\left(X_j\right)^2}{n}\right]\\ & = \frac{1}{n}\sum_{j = 1}^{n}\operatorname{E}\left[\left(X_j\right)^2\right]\\ & = \frac{1}{n}\sum_{j = 1}^{n}\left(\sigma^2 + \mu^2\right)\\ & = \sigma^2 + \mu^2 \end{split} \end{equation*}$$ $$\begin{equation*} \begin{split} \implies \operatorname{Var}\left(X_i^{∗b}\right) &= \sigma^2 + \mu^2 - \mu^2\\ &= \sigma^2.\,\square \end{split} \end{equation*}$$