I stumbled upon this article on confidence intervals. The concept as a whole made sense but seemed strange to me. I concluded that given a fixed method of randomly sampling the population, a fixed sample size and a fixed desired confidence level, the more uniformly distributed the population actually is, the larger the confidence interval is likely to be; similarly, if the desired interval length is fixed, the more uniformly distributed the population actually is, the smaller the confidence level is likely to be.

As an example, trying to estimate the percentage of voters for one party out of two, the random sample conforms to a binomial / hypergeometric distribution, in which case the variance increases as the probability approaches 0.5; i.e. the chance of getting a sampled result further from the actual value increases as the original distribution is more even / uniform, and thus the closer the sampled value is to 50%, the less confidence there is.

It makes a lot of sense to me, since in a way this points out that our logic is "biased" towards non-uniform data, and that variations are "easier" to detect. But then, what methods are there to assert that a population's distribution is indeed uniform (other than increasing sample size / number of samples)?

Also, I wonder whether this principle holds for non-binary hypotheses as well; for instance, determining the percentage of voters for each party in an election with more than two parties, or trying to include non-voters in the calculations as well.

  • 1
    $\begingroup$ Because you appear to apply the term "uniform" to distributions in a seemingly unusual way, could you elaborate on what you mean by that term? $\endgroup$ – whuber Mar 15 at 17:50
  • $\begingroup$ @whuber That the population is equally and mutually exclusively distributed between all possible groups. Two parties means 50%-50%, three 33.3%-33.3%-33.3%, etc. $\endgroup$ – Eyal Roth Mar 15 at 18:24
  • $\begingroup$ Perhaps, more generally, you're thinking the variability of the population. If you're looking at a confidence interval for a population proportion, the variance is greatest and the interval is longest when the proportion is around 1/2. // If you're looking at a confidence interval for a normal population mean, the interval is longest when the sample variance is greatest: For a large sample size a typical 95% CI for the population mean μ is roughly $\bar X ±2S/\sqrt{n},$ where $\bar X$ is the sample mean and $S$ is the sample standard deviation (square root of variance) and $n$ is large. $\endgroup$ – BruceET Mar 16 at 4:34
  • $\begingroup$ Also, generally speaking, in any one particular setting, confidence intervals get shorter as the sample size $n$ increases. This holds even for types of confidence intervals not mentioned above. // I (or someone else here) can give you some examples of various confidence intervals if you will be more specific about what data types and parameters interest you most. $\endgroup$ – BruceET Mar 16 at 4:36
  • $\begingroup$ @BruceET Yes, one way to put it is that I'm thinking of the variability of the population. In cases where the population is most varied / uniformly distributed, my confidence would be smaller compared to if the population would have been less varied. Therefore, my estimations of varied populations are less confident, and I'm asking whether there is a better method / procedure to overcome this and be confident in my estimation independent of the population distribution (I'm guessing not?). $\endgroup$ – Eyal Roth Mar 17 at 14:08

I understand that you are trying to understand factors that influence the width of confidence intervals (CIs), but in some respects, you are not going about it in the right way. In this regard, itt is probably best to forget the (somewhat elusive) concep of 'uniformity' ad focus on variability and sample size. In most settings, confidence intervals get narrower with increasing sample size and wider with increasing variability.

CIs for population population mean $\mu$: If data are nearly normal, then a 95% CI for $\mu$ based on $n$ observations is of the form $$\bar X \pm t^*\frac{S}{\sqrt{n}},$$ where $\bar X$ is the sample mean $\bar X = \frac 1 n \sum_{i=1}^n X_i,$ the sample standard deviation is $S = \sqrt{\frac{1}{n-1}\sum_{i=1}^n (X_i - \bar X)^2},$ $n$ is the sample size, and $t^*$ cuts probability .025 from the upper tail of Student's t distribution with $n-1$ degrees of freedom. For $n > 30,$ it turns out that $t^* \approx 2,$ but it is best to get the exact value from printed tables or using software.

The standard deviation of the random variable $\bar X$ is $SD(\bar X) = S/\sqrt{n},$ often called the 'standard error'. The quantity $M = t^*S/\sqrt{n}$ is called the 'margin of error'; it is half the width of the confidence interval. So small variability (reflected in a small $S)$ and large $n$ lead to a relatively small $M$ and hence a relatively short CI.

Examples: (1) Consider a sample of size $n=10$ from a population distributed $\mathsf{Norm}(\mu = 100, \sigma = 15).$ For the particular sample shown below, the 95% CI for $\mu$ is $(94.66, 110.34)$ of length about $15.68.$ [Computations in R.]

x = round(sort(rnorm(10, 100, 15))); x
##  84  93  98  99 102 103 104 108 108 126
CI = t.test(x)$conf.int; CI; diff(CI)
##  94.66184 110.33816
## attr(,"conf.level")
## 0.95
## 15.67631

(2) By contrast (using the same method), a larger sample of s $n = 100$ from the less variable distribution $\mathsf{Norm}(\mu = 100, \sigma=5)$ has 95% CI $(98.89, 101.05)$ 0f length about $2.15.$

CIs for proportion $p$: If we have $x$ Successes in $n$ independent binomial trials, then one style of 95% CI for the true population proportion of Successes is $$\hat p \pm 1.96\sqrt{\hat p(1-\hat p)/\hat n},$$ where $\hat n = n+4$ and $\hat p = (x+2)/\hat n.$

Examples: (1) Independently sampling $n = 25$ people and asking whether they support Proposition A in an upcoming election, we might get $x = 14$ in favor if the population proportion is $p = 1/2.$ Then a (very long, perhaps even useless) 95% CI for $p$ is $(0.349, 0.756)$ of length about 0.41.

set.seed(2019);  n = 25; x = rbinom(1, n, .5); x
## 14
CI = binom.test(x,n)$conf.int; CI; diff(CI)
## 0.3492816 0.7559763
## attr(,"conf.level")
## 0.95
## 0.4066947

(2) By contrast, if we sample $n = 2500$ people from a population win which $p = .55,$ we might get $x = 1386$ in favor. Then (using the same method) the 95% CI is $(.534, .574)$ of length about $0.04.$ Then we have enough information to expect that the Proposition might pass with more than 50% of the vote.

The margin of error here is about $0.02 = 2\%.$ When $p$ is likely to be around $p = 1/2,$ pollsters often use the guideline that a sample of size $n$ gives a margin of error $M \approx 1/\sqrt{n}.$

(3) It is true that the standard error $\sqrt{p(1-p)/n}$ is smaller when $p$ is far from $1/2,$ so you will get shorter confidence intervals for values of $p$ nearer to 0 or 1. [For a given $n,$ the standard error reaches its maximum when $p = 1/2.]$

If 90% of newly assembled computers of a particular type are satisfactory coming off the assembly line (without any adjustments), and we sample $n = 1000$ such computers, then we might find $x = 901$ computers that do not need adjustment. In this case the 95% CI for $p$ is $(.881, .919)$ of length about $.04.$ For such extreme values of $p$ one can get useful confidence intervals with somewhat smaller sample sizes.

In summary, it is difficult to generalize about lengths of confidence intervals because there are so many different kinds of CIs for many different kinds of parameters and distributions. But it is safe to say that increasing the sample size $n$ tends to give shorter intervals. And as the variability of the population becomes greater, confidence intervals for many parameters tend to be longer.

  • $\begingroup$ Thanks, this asserts my understanding of confidence intervals. But again, given that I have a fixed sample size (I only have a finite amount of money to give people for a survey, thus a finite amount of people), if I were to use confidence intervals, I would be more or less confident in my estimation depending on the variability of the population. I want to be as confident in my estimations regardless of the variability of the population (it is after all what I'm trying to estimate). $\endgroup$ – Eyal Roth Mar 17 at 19:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.