33
$\begingroup$

I recently read this passage from a website and I just can't work out the math.

Overall, it says you can be 93.75% confident of having the true median parameter within an interval, obtained from a random sample of 5 out of a 10 000 population.

Could someone guide me to obtain this value? Here's the original passage:

Pretend for a moment that you’re a decision-maker for a large corporation with 10,000 employees. You’re considering automating part of some routine activity, like scheduling meetings or preparing status reports. But you are facing a lot of uncertainty and you believe you need to gather more data. Specifically, one thing you’re looking for is how much time the typical employee spends each day commuting.

How would you gather this data?

You could create what essentially would be a census where you survey each of the 10,000 employees. But that would be very labor-intensive and costly. You probably wouldn’t want to go through that kind of trouble. Another option is to get a sample, but you are unsure what the sample size should be to be useful.

What if you were told that you might get enough information to make a decision by sampling just five people?

Let’s say that you randomly pick five people from your company. Of course, it’s hard for humans to be completely random, but let’s assume the picking process was about as random as you can get.

Then, let’s say you ask these five people to give you the total time, in minutes, that they spend each day in this activity. The results come in: 30, 60, 45, 80, and 60 minutes. From this, we can calculate the median of the sample results, or the point at which exactly half of the total population (10,000 employees) is above the median and half is below the median.

Is that enough information?

Many people, when faced with this scenario, would say the sample is too small – that it’s not “statistically significant.” But a lot of people don’t know what statistically significant actually means.

Let’s go back to the scenario. What are the chances that the median time spent in this activity for 10,000 employees, is between 30 minutes and 80 minutes, the low and high ends, respectively, of the five-employee survey?

When asked, people often say somewhere around 50%. Some people even go as low as 10%. It makes sense, after all; there are 10,000 employees and countless individual commute times in a single year. How can a sample that is viewed as not being statistically significant possibly get close?

Well, here’s the answer: the chances that the median time spent of the population of 10,000 employees is between 30 minutes and 80 minutes is a staggering 93.75%.

In other words, you can be very confident that the median time spent is between 30 minutes and 80 minutes, just by asking five people out of 10,000 (or 100,000, or 1,000,000 – it’s all the same math).

From : https://hubbardresearch.com/two-ways-you-can-use-small-sample-sizes-to-measure-anything/

$\endgroup$
7
  • 35
    $\begingroup$ I have a soft spot in my heart for this way of finding CIs for the median. But in addition to the good answers already posted below, be sure to note that "we are 93.75% confident that the median is between 30 and 80 minutes" does NOT mean that "93.75% of our employees' commute times are between 30 and 80 minutes". If you wanted a reasonable estimate of that, you'd need quite a bit more than 5 observations. Also, "the median is between 30 and 80 minutes" is so wide that it's probably not of much practical use to your corporate decision-maker. $\endgroup$
    – civilstat
    Mar 29, 2023 at 1:44
  • 3
    $\begingroup$ This is more or less the perfect description of getting ballpark figures. $\endgroup$
    – David S
    Mar 29, 2023 at 14:32
  • $\begingroup$ "Well, here’s the answer: the chances that the median time spent of the population of 10,000 employees is between 30 minutes and 80 minutes is a staggering 93.75%." So you already got the answer? DRTL what is the question? $\endgroup$ Mar 29, 2023 at 14:42
  • 4
    $\begingroup$ It's worth pointing out that, while this is a valid procedure for getting a confidence interval the passage makes the common mistake of misinterpreting the confidence interval as a statement about the parameter of interest rather than a statement about the random interval. From the usual Frequentist perspective, the median is either 100% between 30 and 80 or 0% between 30 and 80. But many intervals constructed using this procedure will catch the true median with the stated probability (up-to minor inaccuracies due to finiteness of the population and discreteness of the times). $\endgroup$
    – guy
    Mar 29, 2023 at 16:06
  • 11
    $\begingroup$ Less of a nit-pick is the incorrect reversal of the interpretation at the end: "the chances that the median time spent of the population of 10,000 employees is between 30 minutes and 80 minutes is a staggering 93.75%." That's not supported by analysis or statistical theory. This is more accurately characterized as "the chance that all values in a random sample of this size from a population with a median less than 30 minutes or greater than 80 minutes would all lie between 30 and 80 minutes is at most one in 16 (6.25%)." $\endgroup$
    – whuber
    Mar 29, 2023 at 17:36

3 Answers 3

69
$\begingroup$

Let's ignore the numbers for a bit. If we draw five observations from the population, the probability that all five observations are above the median is $\left({1\over 2}\right)^5 = 1/32 = 0.03125$, and similarly for the probability that all five observations are below the median. As the events "above the median" and "below the median" are mutually exclusive, we can calculate the probability that all five observations are either entirely above the median or entirely below the median as the sum of the probabilities: $0.03125 + 0.03125 = 0.0625$. Consequently, the probability that a sample will "enclose" the median is just $1 - 0.0625 = 0.9375$.

After you've drawn the sample, of course, probabilities don't apply anymore, but you can construct a $93.75\%$ confidence interval for the median in the obvious way by using the largest and smallest observations.

$\endgroup$
4
  • 6
    $\begingroup$ Sidenote: the probability to be above/below the median with 0.5 probability is only true for continuous distributions. It fails when the variable is a discrete variable. Technically speaking; when we observe only minutes, a discrete variable, then the probability to be above/below the median is smaller than $2 \times 0.5^5$. $\endgroup$ Mar 29, 2023 at 14:48
  • 5
    $\begingroup$ The "out of 10,000" value doesn't enter into this computation, and it feels like we would have better confidence having sampled 5 out of a population of 10, than we would having sampled 5 out of a population of 1 billion. Is that just bad intuition? $\endgroup$
    – Ralph J
    Mar 31, 2023 at 11:44
  • 2
    $\begingroup$ @RalphJ: Your intuition is right. The calculation in this answer is based on “sampling with replacement” — you randomly pick a name five times, and if John Doe’s name comes up three times, you include him three times in your sample. “Sampling without replacement” — randomly picking five distinct names — will always be at least as good in its estimation of the median. If the population is large then the improvement is negligible (the chances of picking the same person twice were tiny anyway); if the population is small, the improvement is significant. $\endgroup$
    – PLL
    Mar 31, 2023 at 12:55
  • 1
    $\begingroup$ @RalphJ - you're quite right. My objective was to explain where the 93.75% came from; many people in more introductory level classes handwave over the distinction between "sampling w/o replacement from a large population" and "sampling from a continuous distribution" in their desire to come up with a concrete, real-world-ish example - and it's that calculation that is the crux of the question, after all. $\endgroup$
    – jbowman
    Mar 31, 2023 at 14:17
24
$\begingroup$

Yes, this really works, under certain conditions, with a couple of caveats

  1. Random selection: You can't just ask any 5 people. It would need to be randomly selected from the population whose median you wanted an interval for.

  2. Understanding what a confidence interval means. The interval for a parameter will have a certain coverage ... but that doesn't necessarily correspond to how confident you personally are about it ... personal confidence is not the same thing as coverage.

    Specifically, that 93.75 percent is a frequentist probability - a long run proportion. Loosely, if you use the same methodology many, many times, about 93.75 percent of those intervals will include the population median.

  3. The calculation of the coverage is based on assuming continuous responses.

  4. It's not necessarily very useful; the range of 5 values will tend to be quite wide.

The calculation of the coverage is mathematically straightforward (see the last paragraph below) but it's also easy to see via simulation. e.g. here's a quick simulation in R:

 mean(replicate(1000000,between(range(runif(5)),0.5)))
 [1] 0.937464

(where between is just: function(x, m) x[1]<m & x[2]>m; if you were doing it for a discrete variable you'd want <= and >= and to define your interval to be closed; it doesn't matter in the continuous case)

It doesn't really matter how big the population was; this calculation effectively uses infinite population. A small population would not have a lower chance.

I used the uniform distribution as a source of continuous random numbers but the same result would apply for any other continuous distribution, since the order relationships are unaltered by any monotonic transformation.

With a continuous variable, the probability that all the values lay to the left of the population median would be $\frac12^5 = \frac{1}{32}$. Similarly for them all to be to the right. Consequently the coverage of the range of 5 randomly selected values is $\frac{15}{16} = 0.9375$.

$\endgroup$
21
$\begingroup$

The other answers have this exactly correct, but I'll explain why it seems so surprising. The trick is that the way the problem is posed hides the goalposts a little bit. We know we have a tiny sample and a high-confidence CI, but the problem sort of glosses over the fact that when choosing even just 5 individuals, the width of the "max-min" range will usually be quite large. It should not be terribly surprising that we can confidently claim that the median is within some very large range. We are likely treading into the territory of "statistically significant, but practically useless". Even very small samples can be used to make conclusions of arbitrary statistical confidence simply by relaxing the width of the tested interval. Here, the sampling approach naturally gives us a large interval, which might be the surprising part.

A knee-jerk reaction might be to think that a small sample size and high-confidence CI are incompatible and cannot be observed together. But given any sample size at all, you can build a CI of any confidence you want, so long as you make it wide enough. What's surprising here is just how wide of a range you get, on average, when selecting only 5 individuals from the population. Choosing 5 individuals from any distribution at all results in a range that covers, on average, the middle two thirds of the population! And since this method tends to put the range nearer the middle than the extremes of possible values, the chance of containing the median individual is even higher than percentage of the population covered.

With that knowledge, it shouldn't be surprising to define a range using a method that usually covers a majority of the population, and be quite confident that the median is in that range. Yes, we have a method that reliably generates a range that contains the median, but that range is so large that it usually contains most other observed values, too. It's already unlikely to pick 5 individuals and find a range that covers less than 50 percentiles of the population, and even less likely to have that sub-majority range land entirely on one side of the median, which is the only way you can avoid containing the median.

$\endgroup$
6
  • 3
    $\begingroup$ This is a surprising side-effect to this question. Indeed, any sample can give you some a% confidence. Whether the sample is size 5 or not, there will always be an a% confidence interval. A sample of two can also give an a% confidence interval. We can be 93.75% confident from any sample. $\endgroup$ Mar 29, 2023 at 15:17
  • 1
    $\begingroup$ I believe a more standard terminology in statistics (contrary to biology) is a sample of 5 rather than 5 samples. The latter suggest 5 samples of some (unknown?) size. $\endgroup$ Mar 29, 2023 at 19:12
  • 1
    $\begingroup$ @RichardHardy You nailed it, my biology background is showing. Changed to "individuals" rather than "samples". $\endgroup$ Mar 30, 2023 at 13:10
  • 1
    $\begingroup$ This is an excellent addendum to the other answers. Another way to think about this is as an example of the distinction between accuracy and precision. You can generally increase your accuracy by decreasing your precision. E.g.: if I say a given individual is in country X, it's much more likely to be correct than saying they are in a specific city at a given moment. More correct but much less useful if we are trying to find said individual. $\endgroup$
    – JimmyJames
    Mar 30, 2023 at 17:08
  • $\begingroup$ @SextusEmpiricus With only one observation, the only way to get a 95% CI is by taking the entire codomain. $\endgroup$ Apr 1, 2023 at 0:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.