Can you be 93.75% confident from a random sample of only five from a population of 10 000?

Question

I recently read this passage from a website and I just can't work out the math.

Overall, it says you can be 93.75% confident of having the true median parameter within an interval, obtained from a random sample of 5 out of a 10 000 population.

Could someone guide me to obtain this value? Here's the original passage:

Pretend for a moment that you’re a decision-maker for a large corporation with 10,000 employees. You’re considering automating part of some routine activity, like scheduling meetings or preparing status reports. But you are facing a lot of uncertainty and you believe you need to gather more data. Specifically, one thing you’re looking for is how much time the typical employee spends each day commuting.

How would you gather this data?

You could create what essentially would be a census where you survey each of the 10,000 employees. But that would be very labor-intensive and costly. You probably wouldn’t want to go through that kind of trouble. Another option is to get a sample, but you are unsure what the sample size should be to be useful.

What if you were told that you might get enough information to make a decision by sampling just five people?

Let’s say that you randomly pick five people from your company. Of course, it’s hard for humans to be completely random, but let’s assume the picking process was about as random as you can get.

Then, let’s say you ask these five people to give you the total time, in minutes, that they spend each day in this activity. The results come in: 30, 60, 45, 80, and 60 minutes. From this, we can calculate the median of the sample results, or the point at which exactly half of the total population (10,000 employees) is above the median and half is below the median.

Is that enough information?

Many people, when faced with this scenario, would say the sample is too small – that it’s not “statistically significant.” But a lot of people don’t know what statistically significant actually means.

Let’s go back to the scenario. What are the chances that the median time spent in this activity for 10,000 employees, is between 30 minutes and 80 minutes, the low and high ends, respectively, of the five-employee survey?

When asked, people often say somewhere around 50%. Some people even go as low as 10%. It makes sense, after all; there are 10,000 employees and countless individual commute times in a single year. How can a sample that is viewed as not being statistically significant possibly get close?

Well, here’s the answer: the chances that the median time spent of the population of 10,000 employees is between 30 minutes and 80 minutes is a staggering 93.75%.

In other words, you can be very confident that the median time spent is between 30 minutes and 80 minutes, just by asking five people out of 10,000 (or 100,000, or 1,000,000 – it’s all the same math).

Answer 1

Let's ignore the numbers for a bit. If we draw five observations from the population, the probability that all five observations are above the median is $\left({1\over 2}\right)^5 = 1/32 = 0.03125$, and similarly for the probability that all five observations are below the median. As the events "above the median" and "below the median" are mutually exclusive, we can calculate the probability that all five observations are either entirely above the median or entirely below the median as the sum of the probabilities: $0.03125 + 0.03125 = 0.0625$. Consequently, the probability that a sample will "enclose" the median is just $1 - 0.0625 = 0.9375$.

After you've drawn the sample, of course, probabilities don't apply anymore, but you can construct a $93.75\%$ confidence interval for the median in the obvious way by using the largest and smallest observations.

Answer 2

Yes, this really works, under certain conditions, with a couple of caveats

1. Random selection: You can't just ask any 5 people. It would need to be randomly selected from the population whose median you wanted an interval for.

2. Understanding what a confidence interval means. The interval for a parameter will have a certain coverage ... but that doesn't necessarily correspond to how confident you personally are about it ... personal confidence is not the same thing as coverage.

   Specifically, that 93.75 percent is a frequentist probability - a long run proportion. Loosely, if you use the same methodology many, many times, about 93.75 percent of those intervals will include the population median.

3. The calculation of the coverage is based on assuming continuous responses.

4. It's not necessarily very useful; the range of 5 values will tend to be quite wide.

The calculation of the coverage is mathematically straightforward (see the last paragraph below) but it's also easy to see via simulation. e.g. here's a quick simulation in R:

 mean(replicate(1000000,between(range(runif(5)),0.5)))
 [1] 0.937464

(where between is just: function(x, m) x[1]<m & x[2]>m; if you were doing it for a discrete variable you'd want <= and >= and to define your interval to be closed; it doesn't matter in the continuous case)

It doesn't really matter how big the population was; this calculation effectively uses infinite population. A small population would not have a lower chance.

I used the uniform distribution as a source of continuous random numbers but the same result would apply for any other continuous distribution, since the order relationships are unaltered by any monotonic transformation.

With a continuous variable, the probability that all the values lay to the left of the population median would be $\frac12^5 = \frac{1}{32}$. Similarly for them all to be to the right. Consequently the coverage of the range of 5 randomly selected values is $\frac{15}{16} = 0.9375$.

A note on the finite-population correction: at n=5 and N~10000 it would make essentially no difference. So just dealing with infinite N suffices here

Answer 3

The other answers have this exactly correct, but I'll explain why it seems so surprising. The trick is that the way the problem is posed hides the goalposts a little bit. We know we have a tiny sample and a high-confidence CI, but the problem sort of glosses over the fact that when choosing even just 5 individuals, the width of the "max-min" range will usually be quite large. It should not be terribly surprising that we can confidently claim that the median is within some very large range. We are likely treading into the territory of "statistically significant, but practically useless". Even very small samples can be used to make conclusions of arbitrary statistical confidence simply by relaxing the width of the tested interval. Here, the sampling approach naturally gives us a large interval, which might be the surprising part.

A knee-jerk reaction might be to think that a small sample size and high-confidence CI are incompatible and cannot be observed together. But given any sample size at all, you can build a CI of any confidence you want, so long as you make it wide enough. What's surprising here is just how wide of a range you get, on average, when selecting only 5 individuals from the population. Choosing 5 individuals from any distribution at all results in a range that covers, on average, the middle two thirds of the population! And since this method tends to put the range nearer the middle than the extremes of possible values, the chance of containing the median individual is even higher than percentage of the population covered.

With that knowledge, it shouldn't be surprising to define a range using a method that usually covers a majority of the population, and be quite confident that the median is in that range. Yes, we have a method that reliably generates a range that contains the median, but that range is so large that it usually contains most other observed values, too. It's already unlikely to pick 5 individuals and find a range that covers less than 50 percentiles of the population, and even less likely to have that sub-majority range land entirely on one side of the median, which is the only way you can avoid containing the median.