1
$\begingroup$

It is common to use, explore and test differences in attributes between groups. While often people are interested in significant differences and large effect sizes, I was wondering about the effect of slight differences between groups with theoretically similar distributions. Specifically, I am interested in how slight changes to a mean impact the number of extreme values (however decided upon).

Ex. Lets assume hypothetically that are measuring the IQ of two groups, A ($M=100, s.d.=15$) and B ($M=102, s.d.=15$). Suppose each group has $N=250$. We can simulate this easily:

set.seed(1)
group_a <- rnorm(250, 100, 15)
group_b <- rnorm(250, 102, 15)

> sum(group_a>145)
[1] 0
> sum(group_b>145)
[1] 1
> t.test(group_a, group_b) 
        Welch Two Sample t-test

data:  group_a and group_b
t = -1.4821, df = 493.35, p-value = 0.1389
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -4.6843958  0.6559131
sample estimates:
mean of x mean of y 
 100.3325  102.3468 

We can see that the difference is non-significant and that group B have one person with an IQ higher than 145. This may be important as somethings correlate with very high attribute values (being a CEO, math prodigy, whathaveyou). If I wish to estimate the difference in the population in very high values of an attribute (e.g., IQ) based on a sample, how do I do it?

To make this clearer, repeating the same code above, but with group sizes of 250,000, the first group has 321 observations above 145, and the second 499 - a whopping $55.4$% more.

So:

  1. What is a statistical explanation of how small differences in means between similar distributions cause massive differences in the far tail?
  2. How will this conclusion change when the standard deviation changes as well?
  3. What can be done to estimate this effect from small sample sizes (e.g., as in the example)?
  4. How will the effect change when the distribution is not normal (lets say with a long right tail)?

Many thanks!

$\endgroup$
1
$\begingroup$

This is fairly strightforward. Using your example,
A (M=100,s.d.=15) and B (M=102,s.d.=15)

  1. Explanation

x = 145 is 3 standard deviations above the mean for distribution A,
but only 3 - 2/15 standard deviations above the mean for distribution B.

p(x>145 | A) = 1 - pnorm(3)      = 0.001349898
p(x>145 | B) = 1 - pnorm(3-2/15) = 0.002074098

0.002074098/0.001349898 = 1.536485

So we would expect distribution B to have 53.6% more points over 145 than distribution A, which matches your simulation pretty well.

This far away from the mean, the curve goes down rather steeply, so even that small difference in the width makes a big difference in the area.

Tail of the normal curve

  1. Changing standard deviation

You do not specify how the standard deviation will change, but you can use the above computation to see the effect.

If distribution A has mean $μ_1$ and standard deviation $σ_1$
and distribution B has mean $μ_2$ and standard deviation $σ_2$
and we look at some threshold T, then

p(x>T | A) = 1 - pnorm(T, $μ_1$, $σ_1$)
p(x>T | B) = 1 - pnorm(T, $μ_2$, $σ_2$)

You can multiply by sample sizes to get the expected number of points that satisfy the condition.

  1. Small Samples

Of course, the probability calculation is exactly the same. That will give an expected percentage of samples above the threshold. In your example, the small sample used N=250.
For distribution A the expected number of points over 145 is 0.001349898 * 250 = 0.3374745
For distribution B the expected number of points over 145 is 0.002074098 * 250 = 0.5185245
It is not surprising that distribution A produced no points and distribution B produced one.

  1. Not normal

Of course, it depends on the distribution. If you know (or are willing to assume) the distributions, you can make probability calculations like the ones above. Without assumptions on the distributions, I don't see how you can say much.

$\endgroup$

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.