Motivation for Confidence Intervals

Question

I am using the following example to motivate the use of confidence intervals and to better understand them. However, I am struggling with some basic definitions and would appreciate some guidance. I would also appreciate if you could point out any inaccuracies that you may find.

The example

Suppose that there are two candidates, candidate A and candidate B, running for an election, and suppose that you want to estimate the probability that candidate A will win, denoted as $p_A$. Suppose that a total of 100,000 people, also known as the population, can vote in this election. You could try and count the number of people in the population who will vote for candidate A, and then divide this by 100,000. However, this is not practical. Instead, you can select a small subset of the population, called a sample, and count how many of them will vote for candidate A and divide that by the sample size. This is called a point estimate of $p_A$. Suppose that you select a sample of 1,000 people.

Let $X_i=1$ if the $i^{th}$ person in the sample will vote for candidate A, and let $X_i=0$ if the $i^{th}$ person in the sample will vote for candidate B, for $i = 1,2,...,1000$. Since the sample's voting choices are random, then $X_1,X_2,...,X_{1000}$ are all random variables. Since voters only have 2 choices, then these are Bernoulli random variables. Additionally, since it can be assumed that voters think similarly and that one voter's choice does not affect another voter's choice, then these random variables are independent and identically distributed with parameter $p_A$. Note that $p_A = P(X_i = 1)$. For example, after picking a random sample of 1,000 people and asking each of them who they will vote for, the observed voting choices could be:

$$\mathcal{D} = \{X_1 = 1,X_2 = 0,X_3 = 1,...,X_{1000} = 1\}$$

The maximum likelihood estimate for $p_A$ can then be computed by summing all the $X_i$'s for $i=1,2,...,1000$ and then dividing this summation by 1000. Mathematically, if $\hat{\Theta}=f(\mathcal{D},1000)$ is a point estimator of $p_A$, then:

$$\hat{\Theta} = \frac{\sum\limits_{i = 1}^{1000} X_i}{1000}$$

More generally, if $\mathcal{D}$ is a sample of size $n$, then:

$$\hat{\Theta} = \frac{\sum\limits_{i = 1}^{n} X_i}{n}$$

For example, if 800 out of 1,000 people said that they would vote for candidate A, then $\hat{\Theta} = 0.8$.

Note, however, that since $\hat{\Theta}$ is a function of random variables, then it is also a random variable. This is because if you select a different sample of 1,000 people from the population, you may observe a different set of voting choices. For example, a new sample chosen from the population could be:

$$\mathcal{D} = \{X_1 = 1,X_2 = 0,X_3 = 0,...,X_{1000} = 0\}$$

So, although $\hat{\Theta}$ is a point estimator, it will have different values for different samples. If a different sample of 1,000 people is taken from the population an infinite number of times and $\hat{\Theta}$ is computed each time, then, theoretically, the true value of $p_A$ is the mean of $\hat{\Theta}$:

$$p_A = E\left[\hat{\Theta}\right]$$

However, it is not practically possible to sample 1,000 different people an infinite number of times. You could try to estimate $E\left[\hat{\Theta}\right]$ using the sample mean. For example, you could sample $n$ different people $k$ times, and each time you compute $\hat{\Theta}$. Then, the estimate of $E\left[\hat{\Theta}\right]$ is:

$$E\left[\hat{\Theta}\right] \approx \frac{\sum_{i=1}^k \hat{\Theta}_i}{k}$$

However, this estimate of $E\left[\hat{\Theta}\right]$ would still be a random variable that depends on the random sample of people and the number of times $k$ that you sample. This does not solve the original problem. Instead, it would be helpful to compute an interval of values centered around the true value $p_A = E\left[\hat{\Theta}\right]$ that $\hat{\Theta}$ could possibly take. This is called interval estimation.

Since $\hat{\Theta}$ is random, then it has an associated probability distribution $p\left(\hat{\Theta}\right)$ called the sampling distribution. In this case, the sampling distribution is dependent on the sample size $n$, such that $p\left(\hat{\Theta}\right) = p\left(\hat{\Theta};n\right)$. Looking closer at the definition of $\hat{\Theta}$:

$$\hat{\Theta} = \frac{\sum\limits_{i = 1}^{1000} X_i}{1000}$$

Since $\sum\limits_{i = 1}^{1000} X_i$ is a sum of i.i.d Bernoulli random variables, then:

$$Y = \sum\limits_{i = 1}^{1000} X_i$$

Is a binomial random variable with mean equal to:

\begin{align} E[Y] &= E\left[\sum\limits_{i = 1}^{1000} X_i\right] \\ &= \sum\limits_{i = 1}^{1000} E[X_i] \\ &= \sum\limits_{i = 1}^{1000} \mu = 1000\mu \end{align}

Where:

$$E[X_1] = E[X_2] = ... = E[X_{1000}] = \mu$$

So, the mean of $\hat{\Theta}$ is:

$$E\left[\hat{\Theta}\right] = \frac{E[Y]}{1000} = \mu$$

Additionally, the variance of $Y$ is:

$$ Var(Y) = Var(X_1 + X_2 + ... + X_{1000}) $$

Since $X_1,X_2,...,X_{1000}$ are independent and therefore uncorrelated, then the variance of their sum is equal to the sum of their variances:

\begin{equation} Var(X_1 + X_2 + ... + X_{1000}) = Var(X_1) + Var(X_2) + ... + Var(X_{1000}) \end{equation}

So:

\begin{align} Var(X_1) &= E\left[(X_1 - \mu)^2\right] \\ &= E\left[X_1^2 - 2X_1\mu + \mu^2\right] \\ &= E\left[X_1^2\right] - 2 \mu E[X_1] + \mu^2 \end{align}

Using LOTUS:

$$ E[X_1^2] = 0^2\cdot p(X_1 = 0) + 1^2\cdot p(X_1 = 1) = p(X_1 = 1) = \mu $$

So: $$ Var(X_1) = \mu - 2\mu^2 + \mu^2 = \mu - \mu^2 = \mu(1-\mu) $$

Since:

$$ Var(Y) = \sum_{i=1}^{1000} Var(X_i) $$

Then:

$$ Var(Y) = 1000\mu(1-\mu) $$

Assuming a large number of samples $n$ are available (e.g. $n=1000$), then $Y$ can instead be approximated by the following normal distribution:

$$Y \sim \mathcal{N}(n\mu,n\mu(1-\mu))$$

Since:

$$\hat{\Theta} = \frac{Y}{1000}$$

And for any random variable $W$:

$$Var(aW) = a^2Var(W)$$

Then:

\begin{align} Var\left(\hat{\Theta}\right) &= \frac{Var(Y)}{1000^2} \\ &= \frac{1000\mu(1-\mu)}{1000^2} \\ &= \frac{\mu(1-\mu)}{1000} \end{align}

More generally, given a sample of $n$ i.i.d Bernoulli random variables each with a mean of $\mu$, and the point estimator $\hat{\Theta}$ computes the sample mean, then:

\begin{align} E\left[\hat{\Theta}\right] &= \mu \\ Var\left(\hat{\Theta}\right) &= \frac{\mu(1-\mu)}{n} \\ \hat{\Theta} &\sim \mathcal{N}\left(\mu,\frac{\mu(1-\mu)}{n}\right) \end{align}

The standard deviation of $\hat{\Theta}$ is called the standard error $\sigma_{\hat{\Theta}}$:

\begin{align} \sigma_{\hat{\Theta}} &= \sqrt{Var\left(\hat{\Theta}\right)} \\ &= \sqrt{\frac{\mu(1-\mu)}{n}} \\ &= \frac{\sqrt{\mu(1-\mu)}}{\sqrt{n}} \\ &= \frac{\sigma_X}{\sqrt{n}} \end{align}

Where $\sigma_X$ is the standard deviation of $X_1,X_2,...,X_n$. Now, recall that the true value of $p_A$ is $E\left[\hat{\Theta}\right] = \mu$, which cannot be practically computed. Instead, the goal is to compute $a$ and $b$ such that:

$$p\left(E\left[\hat{\Theta}\right] - a<\hat{\Theta}<E\left[\hat{\Theta}\right] + b\right) = c$$

Which means that if you sample $n$ people from the population an infinite number of times, and each time you compute $\hat{\Theta}$, $\hat{\Theta}$ will be a maximum of $a$ or $b$ away from the true value $c\%$ of the time. In other words, what values should be chosen for $a$ and $b$ such that $\hat{\Theta}$ lies in the interval $\left[E\left[\hat{\Theta}\right] - a,E\left[\hat{\Theta}\right] + b\right]$ with $c\%$ confidence?

My questions

• Are there any mistakes in what I said so far?
• Is my definition of confidence intervals correct so far? I am basically saying that if I sample $n$ people from the population an infinite number of times, and each time I compute $\hat{\Theta}$, then $\hat{\Theta}$ will lie in the interval $\left[E\left[\hat{\Theta}\right] - a,E\left[\hat{\Theta}\right] + b\right]$ $c\%$ of the time.
• I am not sure how to proceed beyond this point to find $a$ and $b$, so would appreciate some help here. I am assuming that $\hat{\Theta}$ is a Normal random variable as mentioned above.

Answer 1

Your argument seems mostly OK. I'm not sure what you mean by "[S]ince it can be assumed that voters think similarly...." and I'm not sure I follow all of your steps toward the end.

I think you are describing the Wald confidence interval, which is of the form $$\hat \theta \pm 1.96\sqrt{ \frac{\hat\theta(1-\hat\theta) }{n} }.$$ This kind of interval was originally proposed as an 'asymptotic' interval to be use for very large $n.$

The number 1.96 comes from assuming that $n$ is sufficiently large that $\hat \theta = \frac{X}{n}\stackrel{aprx}{\sim}\mathsf{Norm}\left(\mu = \hat\theta, \sigma=\sqrt{ \frac{\hat\theta(1-\hat\theta) }{n} }\right),$ where $X$ is the number of Successes.

There are two approximations involved in the displayed relationship: (a) that $\hat \theta$ is approximately normal and (b) that the standard deviation $\sigma=\sqrt{\frac{\theta(1-\theta)}{n}}$ is well-estimated by $\hat\sigma=\sqrt{\frac{\hat\theta(1-\hat\theta)}{n}}.$ Assumption (a) is reasonable for $\theta$ not too close to $0$ or $1$ and $n$ is at least several hundred, as in most election polls. However, (b) is not reliably accurate unless $n$ is a thousand or more.

For 95% confidence intervals, the Agresti-Coull variation of the Wald confidence interval has been shown to have 95% coverage of $\theta$ more reliably than the Wald interval. It uses $\tilde\theta=\frac{X+2}{n+4}$ and is of the form $\tilde \theta \pm 1.96\sqrt{ \frac{\tilde\theta(1-\tilde\theta) }{n+4} }.$

Rationale: A test for $H_0: \theta=\theta_0$ vs. $H_a: \theta\ne\theta_0$ rejects at level 5% when $|Z| = |(\hat\theta - \theta_0)/\sigma_0| \ge 1.96,$ with $\sigma_0 = \sqrt{\theta_0(1-\theta_0)/n}.$ "Inverting this test" to find the interval of values $\theta_0$ for which $H_0$ is not rejected, we would get a 95% CI for $\theta,$ without using $\hat\sigma$ to estimate $\sigma.$ This involves solving a somewhat messy quadratic equation. By taking $1.96 = 2$ and ignoring small terms in the solution, one gets the Agresti-Coull interval as a good approximation.

Example: In your example with $X=800, n=1000,$ the Wald CI $(0.7752,0.8248)$ the Agresti-Coull CI $(0.777,8266)$ are nearly the same. However, for $X = 80, n = 100,$ the respective intervals $(0.722,0.878)$ and $(0.710, 9.867)$ are noticeably different.

Another good frequentist CI, called the Jeffreys CI, for $\theta$ uses a Bayesian argument with noninformative prior distribution $\mathsf{Beta}(.5,.5).$ It uses the quantiles $0.025$ and $0.975$ of $\mathsf{Beta}(x+.5,n-x+.5).$ In the previous example, the CIs for samples $n=1000$ and $n=100$ are $(0.774,0.824)$ and $(0.714,0.869).$ respectively, in close agreement with the Agresti-Coull intervals above.

qbeta(c(.025,.975), 800.5, 200.5)
[1] 0.7743623 0.8239041

qbeta(c(.025,.975), 80.5, 20.5)
[1] 0.7137204 0.8692096

Simulation: Among a million samples of size $n=50$ with $\theta = 0.81,$ only about 91.3% (not 95%) produced Wald "95% CIs" covering $\theta.$

set.seed(2020)
n = 50;  th = .81
x = rbinom(10^6, n, .8);  th.est = x/n
se = sqrt(th.est*(1-th.est)/n)
mean(th > th.est-1.96*se & th < th.est+1.96*se)
[1] 0.913417

References: The Wikipedia article on binomial confidence intervals discusses these and other styles of confidence intervals. You might also be interested in reading the article by Brown, Cai, and DasGupta (2001), Statistical Science. And Perhaps this Q&A on coverage probabilities of various styles of CIs.