Why is 600 out of 1000 more convincing than 6 out of 10?

Question

Look at this excerpt from "The study skills handbook", Palgrave, 2012, by Stella Cottrell, page 155:

Percentages Notice when percentages are given.
Suppose, instead, the statement above read:

60% of people preferred oranges; 40% said they preferred apples.

This looks convincing: Numerical quantities are given. But is the difference between 60% and 40% significant? Here we would need to know how many people were asked. If 1000 people were asked of whom 600 preferred oranges, the number would be persuasive. However, if only 10 people were asked, 60% simply means 6 people preferred oranges. "60%" sounds convincing in a way that "6 out of 10" does not. As a critical reader, you need to be on the lookout for percentages being used to make insufficient data look impressive.

What is this characteristic called in statistics? I would like to read more about it.

Answer 1

I would like to list another intuitive example.

Suppose I tell you I can predict the outcome of any coin flip. You do not believe and want to test my ability.

You tested 5 times, and I got all of them right. Do you believe I have the special ability? Maybe not. Because I can get all of them right by chance. (Specifically, suppose the coin is a fair coin, and each experiment is independent, then I can get all rights with $0.5^5\approx0.03$ with no super power. See Shufflepants's link for a joke about it).

On the other hand, if you tested me large number of times, then it is very unlikely that I can get it by chance. For example, if you tested $100$ times, the probability of me getting all of them right is $0.5^{100}\approx 0$.

---

The statistical concept is called statistical power, from Wikipeida

The power of a binary hypothesis test is the probability that the test correctly rejects the null hypothesis (H0) when the alternative hypothesis (H1) is true.

Back to the super power on coin flip example, essentially you want to run a hypothesis testing.

• Null hypothesis (H0): I do not have the super power
• Alternative hypothesis (H1): I have the super power

Now as you can see in the numerical example (test me 5 times vs test me 100 times), The statistical power has been affected by the sample size.

More to to read here. (more technical and based on t-test).

An interactive tool to understand statistical power can be found here. Note, the statistical power changes with the sample size !

Answer 2

Think about it in terms of proportions. Let's say that preferring an orange is a success, while preferring an apple is a failure. So your mean success rate is $\mu = \frac{\text{# of sucesses}}{n}$ or in this case .6

The standard error of this quantity is estimated to be $\sqrt{\frac{\mu(1-\mu)}{n}}$. For a small sample size (i.e. 10), the standard error is $\approx .155$ but for a sample size of 1000, the standard error is $\approx .0155$. So basically, as was mentioned in the comments, "sample size matters."

Answer 3

This concept is a consequence of the law of large numbers. From Wikipedia,

According to the law, the average of the results obtained from a large number of trials should be close to the expected value, and will tend to become closer as more trials are performed.

Results from a small sample may be farther from the expected value than that from a larger sample. And so, as stated in the question, one should be cautious of results calculated from small samples. The idea is also explained pretty well in this youTube video.

Answer 4

We're in the situation of estimating some population quantity by some sample quantity. In this case, we're using sample proportions to estimate population proportions, but the principle is considerably more general.

If you think of all the observations in your sample taking the value $1$ when they have the characteristic of interest ("preferred oranges to apples" in the example) and $0$ when they don't, then the proportion of $1$'s is the same as the mean of the set of $0$ and $1$ values -- so you can readily see that a sample proportion is actually a mean.

As we take larger and larger samples (using random sampling), the sample means will tend to converge to the population mean. (This is the law of large numbers.)

However what we really want to have some idea of is how far out we might be (such as might be represented by the width of a confidence interval for the proportion, or by the margin of error, which is normally half of such a width).

Typically, the more data you have, the less uncertainty you will have about some quantity like a mean -- because the standard deviation of the distribution of the sample mean decreases as you take larger samples. [Imagine taking means of many different samples of size 4. The distribution of those means is less variable than the distribution of the original observations-- the standard deviation should tend to be about half as big. Now if you take means of many different samples of size 400, the standard deviation of that should be much smaller again (about $\frac{_1}{^{20}}$th of the standard deviation of the original observations).

The standard deviation of the distribution of the sample mean is one way to measure the typical distance a sample mean is from the population mean, which is decreasing (it decreases as $\frac{_1}{^\sqrt{n}}$, as in the examples above).

As a result, we're more confident about the accuracy of our estimate when the sample is large -- if we repeated our experiment again, other such means would be close to the current one -- they cluster together more and more tightly, and because (in this case) our estimate is unbiased, they're clustering together around the values we're trying to estimate. A single sample mean becomes more and more informative about where the population mean might be.

Answer 5

A rule of thumb for "counting" statistics, like counting the number of people that like oranges, or counting the number of "clicks" in a Geiger counter due to radioactive decay, is that the margin of error for the count is roughly the square-root of the expected count value. Counting statistics are known are Poisson statistics.

The square root of 6 is 2.4-ish, so the margin of error is about 40% (2.4/6). The square root of 600 is 24-ish, so the margin of error is about 4% (24/600). That is why having counted 600 is more significant that counting 6. The relative error is one-tenth.

I'm being a little sloppy about the definition of margin of error. It's really the 1-sigma value, and is not a hard cut-off, but it is the range where you expect most (68%) of the measurements to lie. So if you expect 6 orange eaters, you would expect a series of polls to give you mostly numbers in the 4 to 8 range, like 6,6,5,6,7,2,4,6,3,5,6,6,7,6,10,8,6,5,6,6,9,3,7,8.

Answer 6

I don't have the name you're looking for, but the issue isn't statistical. Psychologically, the way humans process numbers in our brains give greater weight (authority) to larger numbers than it does to smaller numbers because the magnitude (physical size) is visually as important as the representative value. Thus, 600/1000 appears more credible than 6/10. This is why shoppers prefer to see "10% Off!" for values less than 100 and "Save $10!" for values over 100 (called the "Rule of 100"). It's about how our brains react to perception.

An amazing look into this and similar kinds of phenomena are discussed by Nick Kolenda in his online treatise, "An Enormous Guide to Pricing Psychology".

Answer 7

While the actual margin of error is important, the reason it sounds more convincing is because of a more heuristic (rule of thumb) experience with people. The actual margin of error confirms this heuristic has merit.

If the sample is 6 for, and 4 against, this could be 50/50 if a single person changes their vote, or a single person was recorded in error. There is only two more people on the 6 side. Everybody knows two flakes, everybody knows the sample could be cherry-picked: You only asked waitresses and nobody else. Or you only polled 10 college professors in the halls of a university. Or you asked 10 wealthy people outside of Saks Fifth Avenue.

Even the mathematical margin of error presumes true randomness and doesn't account for selection bias, or self-selection bias, or anything else, people can intuitively understand that.

In contrast, the 600 vs. 400 result has 200 more people on one side than the other, and 100 people would have to change their mind. Those numbers are very hard to come by (but not impossible) by some accident of where you were polling, how you got people to agree, how individuals understood or interpreted the question, etc.

It is more convincing not because of a mathematical proof that it should be, but because we know from experience that crowds of 1000 are much more likely to be diverse in their opinions (on anything) than a group of 10. (unless you secretly did your polling at a political party convention or a KKK rally or something else likely to draw a one-sided crowd).

The math only precisely quantifies what we already know by intuition; that it is easier to randomly encounter one or two stray votes out of 10, than it is to randomly encounter 100 or 200 stray votes out of 1000.

Answer 8

Something that has not been mentioned is to look at the problem from a Bayesian point of view.

In a Bayesian setting, a natural approach to this problem would be to use a Beta-Binomial distribution. You can assume that the probability of someone preferring oranges over apples is $p$, which Beta distributed, and that the observations are binomially distributed with parameter $p$: $$ p \sim \mathrm{Beta}(\alpha, \beta)\\ n_o|p \sim \mathrm{Bin}(n,p). $$

Let's assume that you have no a-priori reason to believe that more people prefer oranges over apples or vice-versa ($\beta=\alpha$) but also that you have no strong opinion about this (weak prior: $\beta=\alpha=1$). The prior distribution of $p$ is therefore uniform $\mathrm{U}(0,1)$.

After collecting answers from $n$ questionnaires about people's preferences, you note that $n_o$ respondents prefer oranges and $n_a=n-n_o$ of them prefer apples.

The posterior distribution of $p$ is: $$ p|n_o,n_a \sim \mathrm{Beta}(n_o+1, n_a+1). $$

While the mode of the posterior of $p$ (i.e. the maximum-a-posteriori) is $n_o/(n_o+n_a)$ regardless of the number of respondents, the distribution itself is very different: it is much more peaked for large $n$ than for small ones.

To give you an idea, this is the posterior with $n_o=6$ and $n_a=4$:

While this is the posterior with $n_o=600$ and $n_a=400$:

How do you read these plots? You can reason as follows: "I observe that 6 out of 10 people (randomly chosen from the population) prefer oranges over apples but could the true underlying probability (for the whole population) be 0.4 or 0.8 instead? Well, according to the first plot this is quite possible." If you do the same for the second plot (i.e. with 1000 respondents), you get that $p=0.4$ or $p=0.8$ are very very unlikely (again, I am assuming the 1000 are IID samples from the population).

Please note that although these plots look similar to david25272's, they represent something very different.

His plots ask the question: "assuming a given value of $p$ known, what is the probability of observing $n_o$ people responding that they prefer oranges over apples?"

My plots answer the question: "assuming that I observe $n_o$ people responding that they prefer oranges over apples, what is the probability distribution of $p$, the probability of people preferring oranges over apples?"

Answer 9

The short answer:

Basically it's more convincing to have 600 out of 1000 than six out of 10 because, given equal preferences it's far more likely for 6 out of 10 to occur by random chance.

Let's make an assumption - that the proportion who preferred oranges and apples are actually equal (so, 50% each). Call this a null hypothesis. Given these equal probabilities the likelihood of the two results are:

• Given a sample of 10 people, there is a 38% chance of randomly getting a sample of 6 or more people who prefer oranges (which is not all that unlikely).
• With a sample of 1000 people there is less than 1 in a billion chance of having 600 or more out of 1000 people prefer oranges.

(For simplicity I am assuming an infinite population from which to draw an unlimited number of samples).

---

A simple derivation

One way to derive this result is to simply list out the potential ways in which people can combine in our samples:

For ten people it's easy:

Consider drawing samples of 10 people at random from an infinite population of people with equal preferences for apples or oranges. With equal preferences it's easy to simply list all the potential combinations of 10 people:

Here's the full list.

r   C (n=10)    p
10  1       0.09766%
9   10      0.97656%
8   45      4.39453%
7   120     11.71875%
6   210     20.50781%
5   252     24.60938%
4   210     20.50781%
3   120     11.71875%
2   45      4.39453%
1   10      0.97656%
0   1       0.09766%
    1024    100%

r is the number of results (people who prefer oranges), C is the number of possible ways of that many people preferring oranges, and p is the resulting discrete probability of that many people preferring oranges in our sample.

(p is just C divided by the total number of combinations. Note that there are 1024 ways of arranging these two preferences in total (i.e. 2 to the power of 10).

• For instance there is only one way (one sample) for 10 people (r=10) to all prefer oranges. The same is true for all people preferring apples (r=0).
• There are 10 different combinations resulting in nine of them preferring oranges. (One different person prefers apples in each sample).
• There are 45 samples (combinations) where 2 people prefer apples, etc, etc.

(In general we talk about n C r combinations of results r from a sample of n people. There are online calculators you can use to verify these numbers.)

This list allows us to gives us the probabilities above using just division. There is a 21% chance of getting 6 people in the sample that prefer oranges (210 of 1024 of the combinations). The chance of getting six or more people in our sample is 38% (the sum of all the samples with six or more people, or 386 out of 1024 combinations).

Graphically, the probabilities look like this:

With larger numbers, the number of potential combinations grows rapidly.

For a samples of just 20 people there are 1,048,576 possible samples, all with equal likelihood. (Note: I have only shown every second combination below).

r    C (n=20)   p
20   1          0.00010%
18   190        0.01812%
16   4,845      0.46206%
14   38,760     3.69644%
12   125,970    12.01344%
10   184,756    17.61971%
8    125,970    12.01344%
6    38,760     3.69644%
4    4,845      0.46206%
2    190        0.01812%
0    1          0.00010%
     1,048,576  100%

There is still only one sample where all 20 people prefer oranges. Combinations that feature mixed results are much more likely, simply because there are many more ways that the people in the samples can be combined.

Samples that are biased are much more unlikely, just because there are fewer combinations of people that can result in those samples:

With just 20 people in each sample, the cumulative probability of having 60% or more (12 or more) people in our sample preferring oranges drops to just 25%.

The probability distribution can be seen to become thinner and taller:

With 1000 people the numbers are huge

We can extend the above examples to larger samples (but the numbers grow too rapidly for it to be feasible to list all the combinations), instead I have calculated the probabilities in R:

r   p (n=1000)
1000    9.332636e-302
900     5.958936e-162
800     6.175551e-86
700     5.065988e-38
600     4.633908e-11
500     0.02522502
400     4.633908e-11
300     5.065988e-38
200     6.175551e-86
100     5.958936e-162
0       9.332636e-302

The cumulative probability of having 600 or more out of 1000 people prefer oranges is just 1.364232e-10.

The probability distribution is now much more concentrated around the center:

[

(For example to calculate probability of exactly 600 out of 1000 people preferring oranges in R use dbinom(600, 1000, prob=0.5) which equals 4.633908e-11, and the probability of 600 or more people is 1-pbinom(599, 1000, prob=0.5), which equals 1.364232e-10 (less than 1 in a billion).

Answer 10

This is because higher number ensures greater accuracy. For ex, if u would pick up 1000 random people from anywhere on the planet and 599 of them are male against 10 random people with 6 male, the former would be more accurate. Similarly, if you assume a population of 7 billion and calculate the number of males, you would get a more precise number which would obviously be more convincing than with just 1000 people.