41
$\begingroup$

Look at this excerpt from "The study skills handbook", Palgrave, 2012, by Stella Cottrell, page 155:

Percentages Notice when percentages are given.
Suppose, instead, the statement above read:

60% of people preferred oranges; 40% said they preferred apples.

This looks convincing: Numerical quantities are given. But is the difference between 60% and 40% significant? Here we would need to know how many people were asked. If 1000 people were asked of whom 600 preferred oranges, the number would be persuasive. However, if only 10 people were asked, 60% simply means 6 people preferred oranges. "60%" sounds convincing in a way that "6 out of 10" does not. As a critical reader, you need to be on the lookout for percentages being used to make insufficient data look impressive.

What is this characteristic called in statistics? I would like to read more about it.

$\endgroup$
  • 38
    $\begingroup$ sample size matters $\endgroup$ – Aksakal Jul 28 '17 at 17:57
  • 36
    $\begingroup$ I pick two people at random, they are both male, and therefore I conclude that 100% of Americans are male. Convincing? $\endgroup$ – Casey Jul 28 '17 at 19:06
  • 2
    $\begingroup$ It's the "Don't compare apples with oranges" principle $\endgroup$ – wolfies Jul 28 '17 at 19:31
  • 2
    $\begingroup$ To approach that question from a different angle you may consider digging the literature of the framing effect. However, it is an example of cognitive bias and is a psychological topic, not statistical. $\endgroup$ – Larx Jul 30 '17 at 2:46
  • 1
    $\begingroup$ You can imagine a difference of 1 how much it will impact the estimated quantity. 7/10 is much further relatively from 6/10 than 601/1000 is from 600/1000. $\endgroup$ – mathreadler Jul 31 '17 at 15:22

10 Answers 10

55
$\begingroup$

I would like to list another intuitive example.

Suppose I tell you I can predict the outcome of any coin flip. You do not believe and want to test my ability.

You tested 5 times, and I got all of them right. Do you believe I have the special ability? Maybe not. Because I can get all of them right by chance. (Specifically, suppose the coin is a fair coin, and each experiment is independent, then I can get all rights with $0.5^5\approx0.03$ with no super power. See Shufflepants's link for a joke about it).

On the other hand, if you tested me large number of times, then it is very unlikely that I can get it by chance. For example, if you tested $100$ times, the probability of me getting all of them right is $0.5^{100}\approx 0$.


The statistical concept is called statistical power, from Wikipeida

The power of a binary hypothesis test is the probability that the test correctly rejects the null hypothesis (H0) when the alternative hypothesis (H1) is true.

Back to the super power on coin flip example, essentially you want to run a hypothesis testing.

  • Null hypothesis (H0): I do not have the super power
  • Alternative hypothesis (H1): I have the super power

Now as you can see in the numerical example (test me 5 times vs test me 100 times), The statistical power has been affected by the sample size.

More to to read here. (more technical and based on t-test).

An interactive tool to understand statistical power can be found here. Note, the statistical power changes with the sample size !

enter image description here

$\endgroup$
  • 24
    $\begingroup$ Obligatory XKCD $\endgroup$ – Shufflepants Jul 28 '17 at 20:57
  • 5
    $\begingroup$ This does not actually answer the question. The question has two parts: "Why is [it] more convincing…" and "What is this characteristic called in statistics?" You provided an example that asks the reader if they are more convinced by the example, but you have not explained the phenomenon, nor attempted to address why it's more convincing (just relying on the reader's intuitive reasoning on a single example). In addition, you haven't touched on the second question: If you believe this phenomenon doesn't have a name, then explicitly state that. $\endgroup$ – Makyen Jul 29 '17 at 17:48
  • 1
    $\begingroup$ @Mayken That is plainly an attempted answer. It's not the greatest answer and it's not a complete answer but partial or socratic answers are not grounds for claiming it's not an answer at all. Questions should ask one question and if they ask a series of questions I think it's perfectly reasonable (if less than ideal) to only address some of them. If you feel it's a poor answer, you're certainly free to address the issues you see as problems. [For that matter I don't think I answered the second question either; I only discussed the effect; not every phenomenon will have an explicit name] $\endgroup$ – Glen_b Jul 30 '17 at 7:01
  • 1
    $\begingroup$ I can't help but feel like 100% is a special case and this answer doesn't really support my intuition that 600/1000 is more convincing than 6/10. Even if we take it to not be a special case, this is essentially just restating the problem with a different percentage. $\endgroup$ – NotThatGuy Jul 30 '17 at 23:14
  • 2
    $\begingroup$ @Juya it is a web based tool built by someone. You can try it here $\endgroup$ – Haitao Du Aug 8 '17 at 15:59
29
$\begingroup$

Think about it in terms of proportions. Let's say that preferring an orange is a success, while preferring an apple is a failure. So your mean success rate is $\mu = \frac{\text{# of sucesses}}{n}$ or in this case .6

The standard error of this quantity is estimated to be $\sqrt{\frac{\mu(1-\mu)}{n}}$. For a small sample size (i.e. 10), the standard error is $\approx .155$ but for a sample size of 1000, the standard error is $\approx .0155$. So basically, as was mentioned in the comments, "sample size matters."

$\endgroup$
16
$\begingroup$

This concept is a consequence of the law of large numbers. From Wikipedia,

According to the law, the average of the results obtained from a large number of trials should be close to the expected value, and will tend to become closer as more trials are performed.

Results from a small sample may be farther from the expected value than that from a larger sample. And so, as stated in the question, one should be cautious of results calculated from small samples. The idea is also explained pretty well in this youTube video.

$\endgroup$
  • 5
    $\begingroup$ There are several well-known theorems known in statistics as "laws of large numbers," but none of them make statements that look like the one in the question. How, then, do you establish the connection? $\endgroup$ – whuber Jul 28 '17 at 18:28
  • 3
    $\begingroup$ From wikipedia en.wikipedia.org/wiki/Law_of_large_numbers, "According to the law, the average of the results obtained from a large number of trials should be close to the expected value, and will tend to become closer as more trials are performed". Results from a small sample may be farther from the expected value than that from a larger sample. And so, as stated in the question, one should be cautious of results calculated from small samples. $\endgroup$ – Evan Phibbs Jul 28 '17 at 18:50
  • 6
    $\begingroup$ That's a good explanation, thank you. Although you are correct to write that one should be cautious about just how many a "large number of trials" needs to be for the law to be trusted, the application is intuitively sound. I would like to suggest that your explanation belongs in your answer, where it would be more widely read and appreciated, rather than buried in a comment. Link-only answers (such as your link to Youtube) without explanation don't last here. $\endgroup$ – whuber Jul 28 '17 at 19:02
6
$\begingroup$

We're in the situation of estimating some population quantity by some sample quantity. In this case, we're using sample proportions to estimate population proportions, but the principle is considerably more general.

If you think of all the observations in your sample taking the value $1$ when they have the characteristic of interest ("preferred oranges to apples" in the example) and $0$ when they don't, then the proportion of $1$'s is the same as the mean of the set of $0$ and $1$ values -- so you can readily see that a sample proportion is actually a mean.

As we take larger and larger samples (using random sampling), the sample means will tend to converge to the population mean. (This is the law of large numbers.)

However what we really want to have some idea of is how far out we might be (such as might be represented by the width of a confidence interval for the proportion, or by the margin of error, which is normally half of such a width).

Typically, the more data you have, the less uncertainty you will have about some quantity like a mean -- because the standard deviation of the distribution of the sample mean decreases as you take larger samples. [Imagine taking means of many different samples of size 4. The distribution of those means is less variable than the distribution of the original observations-- the standard deviation should tend to be about half as big. Now if you take means of many different samples of size 400, the standard deviation of that should be much smaller again (about $\frac{_1}{^{20}}$th of the standard deviation of the original observations).

The standard deviation of the distribution of the sample mean is one way to measure the typical distance a sample mean is from the population mean, which is decreasing (it decreases as $\frac{_1}{^\sqrt{n}}$, as in the examples above).

As a result, we're more confident about the accuracy of our estimate when the sample is large -- if we repeated our experiment again, other such means would be close to the current one -- they cluster together more and more tightly, and because (in this case) our estimate is unbiased, they're clustering together around the values we're trying to estimate. A single sample mean becomes more and more informative about where the population mean might be.

$\endgroup$
4
$\begingroup$

A rule of thumb for "counting" statistics, like counting the number of people that like oranges, or counting the number of "clicks" in a Geiger counter due to radioactive decay, is that the margin of error for the count is roughly the square-root of the expected count value. Counting statistics are known are Poisson statistics.

The square root of 6 is 2.4-ish, so the margin of error is about 40% (2.4/6). The square root of 600 is 24-ish, so the margin of error is about 4% (24/600). That is why having counted 600 is more significant that counting 6. The relative error is one-tenth.

I'm being a little sloppy about the definition of margin of error. It's really the 1-sigma value, and is not a hard cut-off, but it is the range where you expect most (68%) of the measurements to lie. So if you expect 6 orange eaters, you would expect a series of polls to give you mostly numbers in the 4 to 8 range, like 6,6,5,6,7,2,4,6,3,5,6,6,7,6,10,8,6,5,6,6,9,3,7,8.

$\endgroup$
3
$\begingroup$

I don't have the name you're looking for, but the issue isn't statistical. Psychologically, the way humans process numbers in our brains give greater weight (authority) to larger numbers than it does to smaller numbers because the magnitude (physical size) is visually as important as the representative value. Thus, 600/1000 appears more credible than 6/10. This is why shoppers prefer to see "10% Off!" for values less than 100 and "Save $10!" for values over 100 (called the "Rule of 100"). It's about how our brains react to perception.

An amazing look into this and similar kinds of phenomena are discussed by Nick Kolenda in his online treatise, "An Enormous Guide to Pricing Psychology".

$\endgroup$
  • 2
    $\begingroup$ Though other answer are not incorrect, I do not see them actually correctly addressing the quoted text as this is. The text is addressing the impact of presenting the numbers and the perceived accuracy, not if the numbers actually are more accurate. That is, that you can make a person perceive that your information is more significant by saying 600 of 1000 or 60% or you can cause is to seem less significant by saying 6 of 10 even though each mean the same thing by implying a smaller sample size without if actually being stated, or even possibly true. $\endgroup$ – dlb Jul 28 '17 at 22:17
  • 1
    $\begingroup$ This example seems fundamentally flawed in my opinion. For example a purchase of 100 gets a %10 discount for a total of 90 while a 100.01 purchase gets a discount of 25 for a total of 75.01 which results is a vastly different price. In fact there is a difference in values until you get up to 250.05(or 250.10 depending on rounding or truncating). The question is talking about sample size and standard error while your example is more about an actual difference because of our perception. $\endgroup$ – Joe W Jul 29 '17 at 14:19
  • $\begingroup$ @Joe W, while I can't speak to the specifics of the "rule of 100" (not being a psychologist), the point my answer is making is that psychologists have determined that people trust a larger number, not because of the larger sample size it represents, but because of the perception of the greater importantance given to a larger number. As an engineer, I'd prefer it not be this way --- but it's the way the human mind works. Should you wish to challenge this, I'd recommend reading the underlying details of Mr. Kolenda's presentation. $\endgroup$ – JBH Jul 29 '17 at 14:53
  • $\begingroup$ You are missing my point, there is a bigger difference in the 10% to 100 and 25 it is not just a matter of perception. The simple fact is if you took both values at 100 there is a 15% difference which is a value that is much simpler to measure then standard error when talking two different samples one of 10 and one of 1000. $\endgroup$ – Joe W Jul 29 '17 at 15:18
  • $\begingroup$ I changed my question to use 10% and $10 to help you out. The point of the "rule of 100" is that people perceive larger numbers as having greater relevance and perceive monetary value as more imporant than a mere percentage. It's an issue somewhat outside the scope of the discussion for the OP's purposes. $\endgroup$ – JBH Jul 29 '17 at 15:20
3
$\begingroup$

While the actual margin of error is important, the reason it sounds more convincing is because of a more heuristic (rule of thumb) experience with people. The actual margin of error confirms this heuristic has merit.

If the sample is 6 for, and 4 against, this could be 50/50 if a single person changes their vote, or a single person was recorded in error. There is only two more people on the 6 side. Everybody knows two flakes, everybody knows the sample could be cherry-picked: You only asked waitresses and nobody else. Or you only polled 10 college professors in the halls of a university. Or you asked 10 wealthy people outside of Saks Fifth Avenue.

Even the mathematical margin of error presumes true randomness and doesn't account for selection bias, or self-selection bias, or anything else, people can intuitively understand that.

In contrast, the 600 vs. 400 result has 200 more people on one side than the other, and 100 people would have to change their mind. Those numbers are very hard to come by (but not impossible) by some accident of where you were polling, how you got people to agree, how individuals understood or interpreted the question, etc.

It is more convincing not because of a mathematical proof that it should be, but because we know from experience that crowds of 1000 are much more likely to be diverse in their opinions (on anything) than a group of 10. (unless you secretly did your polling at a political party convention or a KKK rally or something else likely to draw a one-sided crowd).

The math only precisely quantifies what we already know by intuition; that it is easier to randomly encounter one or two stray votes out of 10, than it is to randomly encounter 100 or 200 stray votes out of 1000.

$\endgroup$
3
$\begingroup$

Something that has not been mentioned is to look at the problem from a Bayesian point of view.

In a Bayesian setting, a natural approach to this problem would be to use a Beta-Binomial distribution. You can assume that the probability of someone preferring oranges over apples is $p$, which Beta distributed, and that the observations are binomially distributed with parameter $p$: $$ p \sim \mathrm{Beta}(\alpha, \beta)\\ n_o|p \sim \mathrm{Bin}(n,p). $$

Let's assume that you have no a-priori reason to believe that more people prefer oranges over apples or vice-versa ($\beta=\alpha$) but also that you have no strong opinion about this (weak prior: $\beta=\alpha=1$). The prior distribution of $p$ is therefore uniform $\mathrm{U}(0,1)$.

After collecting answers from $n$ questionnaires about people's preferences, you note that $n_o$ respondents prefer oranges and $n_a=n-n_o$ of them prefer apples.

The posterior distribution of $p$ is: $$ p|n_o,n_a \sim \mathrm{Beta}(n_o+1, n_a+1). $$

While the mode of the posterior of $p$ (i.e. the maximum-a-posteriori) is $n_o/(n_o+n_a)$ regardless of the number of respondents, the distribution itself is very different: it is much more peaked for large $n$ than for small ones.

To give you an idea, this is the posterior with $n_o=6$ and $n_a=4$:

posterior with n_o=6 and n_a=4

While this is the posterior with $n_o=600$ and $n_a=400$: enter image description here

How do you read these plots? You can reason as follows: "I observe that 6 out of 10 people (randomly chosen from the population) prefer oranges over apples but could the true underlying probability (for the whole population) be 0.4 or 0.8 instead? Well, according to the first plot this is quite possible." If you do the same for the second plot (i.e. with 1000 respondents), you get that $p=0.4$ or $p=0.8$ are very very unlikely (again, I am assuming the 1000 are IID samples from the population).

Please note that although these plots look similar to david25272's, they represent something very different.

His plots ask the question: "assuming a given value of $p$ known, what is the probability of observing $n_o$ people responding that they prefer oranges over apples?"

My plots answer the question: "assuming that I observe $n_o$ people responding that they prefer oranges over apples, what is the probability distribution of $p$, the probability of people preferring oranges over apples?"

$\endgroup$
2
$\begingroup$

The short answer:

Basically it's more convincing to have 600 out of 1000 than six out of 10 because, given equal preferences it's far more likely for 6 out of 10 to occur by random chance.

Let's make an assumption - that the proportion who preferred oranges and apples are actually equal (so, 50% each). Call this a null hypothesis. Given these equal probabilities the likelihood of the two results are:

  • Given a sample of 10 people, there is a 38% chance of randomly getting a sample of 6 or more people who prefer oranges (which is not all that unlikely).
  • With a sample of 1000 people there is less than 1 in a billion chance of having 600 or more out of 1000 people prefer oranges.

(For simplicity I am assuming an infinite population from which to draw an unlimited number of samples).


A simple derivation

One way to derive this result is to simply list out the potential ways in which people can combine in our samples:

For ten people it's easy:

Consider drawing samples of 10 people at random from an infinite population of people with equal preferences for apples or oranges. With equal preferences it's easy to simply list all the potential combinations of 10 people:

Here's the full list.

r   C (n=10)    p
10  1       0.09766%
9   10      0.97656%
8   45      4.39453%
7   120     11.71875%
6   210     20.50781%
5   252     24.60938%
4   210     20.50781%
3   120     11.71875%
2   45      4.39453%
1   10      0.97656%
0   1       0.09766%
    1024    100%

r is the number of results (people who prefer oranges), C is the number of possible ways of that many people preferring oranges, and p is the resulting discrete probability of that many people preferring oranges in our sample.

(p is just C divided by the total number of combinations. Note that there are 1024 ways of arranging these two preferences in total (i.e. 2 to the power of 10).

  • For instance there is only one way (one sample) for 10 people (r=10) to all prefer oranges. The same is true for all people preferring apples (r=0).
  • There are 10 different combinations resulting in nine of them preferring oranges. (One different person prefers apples in each sample).
  • There are 45 samples (combinations) where 2 people prefer apples, etc, etc.

(In general we talk about n C r combinations of results r from a sample of n people. There are online calculators you can use to verify these numbers.)

This list allows us to gives us the probabilities above using just division. There is a 21% chance of getting 6 people in the sample that prefer oranges (210 of 1024 of the combinations). The chance of getting six or more people in our sample is 38% (the sum of all the samples with six or more people, or 386 out of 1024 combinations).

Graphically, the probabilities look like this:

binomial sample size 10

With larger numbers, the number of potential combinations grows rapidly.

For a samples of just 20 people there are 1,048,576 possible samples, all with equal likelihood. (Note: I have only shown every second combination below).

r    C (n=20)   p
20   1          0.00010%
18   190        0.01812%
16   4,845      0.46206%
14   38,760     3.69644%
12   125,970    12.01344%
10   184,756    17.61971%
8    125,970    12.01344%
6    38,760     3.69644%
4    4,845      0.46206%
2    190        0.01812%
0    1          0.00010%
     1,048,576  100%

There is still only one sample where all 20 people prefer oranges. Combinations that feature mixed results are much more likely, simply because there are many more ways that the people in the samples can be combined.

Samples that are biased are much more unlikely, just because there are fewer combinations of people that can result in those samples:

With just 20 people in each sample, the cumulative probability of having 60% or more (12 or more) people in our sample preferring oranges drops to just 25%.

The probability distribution can be seen to become thinner and taller:

binomial sample size 20

With 1000 people the numbers are huge

We can extend the above examples to larger samples (but the numbers grow too rapidly for it to be feasible to list all the combinations), instead I have calculated the probabilities in R:

r   p (n=1000)
1000    9.332636e-302
900     5.958936e-162
800     6.175551e-86
700     5.065988e-38
600     4.633908e-11
500     0.02522502
400     4.633908e-11
300     5.065988e-38
200     6.175551e-86
100     5.958936e-162
0       9.332636e-302

The cumulative probability of having 600 or more out of 1000 people prefer oranges is just 1.364232e-10.

The probability distribution is now much more concentrated around the center:

[binomial sample size 1000[3]

(For example to calculate probability of exactly 600 out of 1000 people preferring oranges in R use dbinom(600, 1000, prob=0.5) which equals 4.633908e-11, and the probability of 600 or more people is 1-pbinom(599, 1000, prob=0.5), which equals 1.364232e-10 (less than 1 in a billion).

$\endgroup$
1
$\begingroup$

This is because higher number ensures greater accuracy. For ex, if u would pick up 1000 random people from anywhere on the planet and 599 of them are male against 10 random people with 6 male, the former would be more accurate. Similarly, if you assume a population of 7 billion and calculate the number of males, you would get a more precise number which would obviously be more convincing than with just 1000 people.

$\endgroup$

protected by kjetil b halvorsen Jul 31 '17 at 11:53

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.