Confidence Interval Questions; and Is that the Best Way

Question

My understanding of confidence intervals (CI) is that you take a sample from a population of size $n$. Then for that particular ($i$th) sample, you calculate its mean, $\bar{X}_i$, and its variance $\sigma_{\bar{X}_i}=\frac{s}{\sqrt{n}}$ where $s$ is the standard deviation (SD) of the sample.

And then to determine, e.g., a $95$% CI you then might assume a normal distribution and calculate a factor $z$ such that $\bar{X}_i\pm z\times \sigma_{\bar{X}_i}$ will encompass $95$% of the area under the presumed normal distribution.

The implication of the notion of CIs is that if you take a multiplicity of samples of size $n$ that $95$% of them will contain the actual mean of the population.

Assuming what I've written is correct, I have several questions:

• Why does the chosen level of the CI play out such that the very same percentage of CIs will contain the mean of the population.

• My larger question is since you are taking a multiplicity of samples and calculating their respective means, why not just make use of the distribution of the sample means, which is much more likely to be normal than any one individual sample.

And finally, is there not then some way, using the parameters estimated from the distribution of the sample means, to claim that $\bar{X}\pm z\times \sigma_{\bar{X}}$, will contain the population mean with $95$% certainty?

In a way, I'm questioning the utility of confidence intervals.

Answer 1

On your first question: Why does the chosen level of the CI play out such that the very same percentage of CIs will contain the mean of the population? This is not always the correct interpretation, so I present more on the underlying varying concepts associated with confidence intervals. To quote from Wikipedia:

• The confidence interval can be expressed in terms of samples (or repeated samples): "Were this procedure to be repeated on numerous samples, the fraction of calculated confidence intervals (which would differ for each sample) that encompass the true population parameter would tend toward 90%."[2]
• The confidence interval can be expressed in terms of a single sample: "There is a 90% probability that the calculated confidence interval from some future experiment encompasses the true value of the population parameter." Note this is a probability statement about the confidence interval, not the population parameter...Here the experimenter sets out the way in which they intend to calculate a confidence interval and to know, before they do the actual experiment, that the interval they will end up calculating has a particular chance of covering the true but unknown value.[4] This is very similar to the "repeated sample" interpretation above, except that it avoids relying on considering hypothetical repeats of a sampling procedure that may not be repeatable in any meaningful sense.
• The explanation of a confidence interval can amount to something like: "The confidence interval represents values for the population parameter for which the difference between the parameter and the observed estimate is not statistically significant at the 10% level".[7] In fact, this relates to one particular way in which a confidence interval may be constructed. In each of the above, the following applies: If the true value of the parameter lies outside the 90% confidence interval, then a sampling event has occurred (namely, obtaining a point estimate of the parameter at least this far from the true parameter value) which had a probability of 10% (or less) of happening by chance.

Also, some important points covered under misunderstanding, to quote further:

Misunderstandings

A 95% confidence level does not mean that for a given realized interval there is a 95% probability that the population parameter lies within the interval (i.e., a 95% probability that the interval covers the population parameter).[13] According to the strict frequentist interpretation, once an interval is calculated, this interval either covers the parameter value or it does not; it is no longer a matter of probability. The 95% probability relates to the reliability of the estimation procedure, not to a specific calculated interval.[14]

Note, there is also Bayesian inference in the form of so-called credible intervals. Per Wikipedia again:

Confidence intervals correspond to a chosen rule for determining the confidence bounds, where this rule is essentially determined before any data are obtained, or before an experiment is done. The rule is defined such that over all possible datasets that might be obtained, there is a high probability ("high" is specifically quantified) that the interval determined by the rule will include the true value of the quantity under consideration. The Bayesian approach appears to offer intervals that can, subject to acceptance of an interpretation of "probability" as Bayesian probability, be interpreted as meaning that the specific interval calculated from a given dataset has a particular probability of including the true value, conditional on the data and other information available. The confidence interval approach does not allow this since in this formulation and at this same stage, both the bounds of the interval and the true values are fixed values, and there is no randomness involved. On the other hand, the Bayesian approach is only as valid as the prior probability used in the computation, whereas the confidence interval does not depend on assumptions about the prior probability.

To answer your larger question, "why not just make use of the distribution of the sample means, which is much more likely to be normal than any one individual sample", you are actually right if the samples are from a uniform distribution. In fact, an approximate way to generate random normal deviates is to average 12 deviates from a uniform distribution. However, a more efficient path is to employ a transformation (employing, for example, a Box-Cox power transformation, see discussion here) is induce normality and not lose so many degrees of freedom. If the data has percentage errors, a log transform is recommended.

Answer 2

Normal data, $\sigma$ known. If $n$ observations are randomly sampled from a normal population with unknown $\mu$ and known $\sigma,$ then a 95% confidence interval for $\mu$ is of the form $\bar X \pm 1.96\frac{\sigma}{\sqrt{n}},$ where $\bar X$ is the sample mean of the observations.

Example: Consider $n = 10$ random observations from $\mathsf{Norm}(100, 15),$ as simulated in R and put into the vector x below. Suppose you know $\sigma=15,$ and estimate $\mu$ by the sample mean $A = \bar X = 98.44.$

set.seed(2020)         # for reprodudibility
x = rnorm(10, 100, 15)
summary(x)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  58.05   86.79  103.14   98.44  109.52  126.39

Then a 95% CI for $\mu$ is $(89.14, 107.74)$. This happens to be one of the 95% of cases in which the CI 'covers' (contails) the population mean $\mu = 100.$

a = mean(x)
CI = a + c(-1.96, 1.96)*15/sqrt(10); CI
[1]  89.14346 107.73765

Normal data, $\sigma$ estimated by $S.$ If $n$ observations are randomly sampled from a normal population with unknown $\mu$ and unknown $\sigma,$ then a 95% confidence interval for $\mu$ is of the form $\bar X \pm t^*\frac{S}{\sqrt{n}},$ where $\bar X$ and $S$ are, respectively, the sample mean and sample stardard deviation, and $t^*$ cuts probability 0.25 from the upper tail of Student's t distribution with $n-1$ degrees of freedom.

Example: Use the same data as above, but now pretend you don't know $\sigma.$ We estimate $\sigma$ by the sample standard deviation $S = 19.36$ and (from R or a printed table of t distributions) $t^* = 2.262.$ Then the expression above gives the 95% CI $(84.59, 112.29).$ [I have used the R function t.test, which prints out the CI.]

s = sd(x);  s
[1] 19.3592
qt(.975, 9)
[1] 2.262157
t.test(x)$conf.int
[1] 84.59182 112.28929
attr(,"conf.level")
[1] 0.95

Nonnormal data. If data are not normal or if you are trying something other than the population mean $\mu,$ then different styles of confidence intervals may be appropriate. If the sample is large enough that $\bar X$ is nearly normal then one of the confidence intervals above may be a useful approximation.

Example: Consider $n=500$ observations from an exponential distribution with rate $\lambda = .01$ and $\mu = \sigma = 100.$ The exponential distribution is severely right-skewed and far from normal. But the average of a large exponential sample is roughly normal. The 95% t confidence interval provides as approximate 95% CI $(93.07, 111.46)$ in this case.

However, it is best to use the exact 95% CI $(93.86, 111.86)$ for $\sigma,$ based on a gamma distribution. Because the gamma distribution is not symmetrical, the exact CI is not centered exactly at $\bar Y = 103.17.$ (In this example, both the approximate CI and the exact CI happen to include the true value $\mu = 100.)$

set.seed(411)
y = rexp(500, .01)
summary(y);  a = mean(y)
    Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
  0.2361  32.3471  73.6159 103.1729 143.8860 596.2742 
[1] 97.17857
t.test(y)$conf.int
[1] 93.07353 111.45858

attr(,"conf.level")
[1] 0.95
a/qgamma(c(.975,.025), 500, 500)
[1]  93.86247 111.85700

Answer 3

since you are taking a multiplicity of samples and calculating their respective means, why not just make use of the distribution of the sample means

Great intuition! Notice that the equations for confidence intervals only require one sample, however. If we could gather many samples of a good size, we would. Usually, it is difficult or impossible to gather more data (we cannot gather more data under the exact conditions that held yesterday, for example), and confidence intervals are one way to quantify uncertainty in this usual case.

Answer 4

Let me fill some gaps:

• Let's take a set of samples (of size $n$) from a single population, and calculate the mean of each sample separately, $\{\overline X_i\}_{i=1,...,m}$ where $m$ is the number of how many different samples we took.

• Since each $\overline X_i$ is calculated from a subset of the population, $\overline X_i$ is a function of the data points in the $i$th sample, but since we have chosen the members of $i$th sample randomly, $\overline X_i$ is a random function.

• We assume that the distribution of each $\overline X_i$ is a normal distribution with the same mean and variance.

• (for an appropriately chosen $z$) %95 of the values $\{\overline X_i\}_{i=1,..m}$ will not be in $\overline X_j \pm z \sigma_j$ because once you took $m$ samples of size $n$ from a single population, those values are not random variables anymore, they are just real numbers. However, the probability of any $\overline X_k$ being in $\overline X_j \pm z \sigma_j$ will be %95. Hence,

Now, why CI is an important concept?

Because it gives you an estimation of your errors. Think about it; for example, if I tell you the distance between two points without telling you how confident I am about the information I gave, how can you use that information (compare $100m \pm 90m$ and $100m \pm 1m$)?