2
$\begingroup$

I apologize in advance as I am not well-versed in statistics, but I hope that this question makes sense.

I have 2 populations which are normally distributed and have a near-identical mean. I would like to know if the upper quantiles (say >0.99) differ - ie, does one population extend further than the other, and how can I see if the difference is significantly different?

One approach I have thought to take is to subset my dataset on just the upper quantiles, then plot the ECDF, but I don't know that this is a correct approach.

Improve this question
$\endgroup$
5
  • $\begingroup$ It depends on whether the two populations have the same distribution. Difficult to know what 'nearly identical' might mean. Then the answer depends on sample sizes; as normal samples become larger, there is more chance to see values far into the right tail. The distribution of the max may be hard to handle, so you should not ask for 99th percentile unless you have a very large sample. Maybe 95th of 97th percentiles would be safer. // Once you find the difference in 95th percentiles for two samples, you could simulate to see if they are unusually far apart. $\endgroup$
    – BruceET
    Commented Jan 6, 2022 at 21:35
  • $\begingroup$ @BruceET Thanks for the response - I agree that as the sample size becomes larger, there is more chance to see values that extend further into the right tail. I've randomly downsample my dataset so that each population has the same number of events (Approx 400,000 events). I am not sure how to test that the distributions are non-identical. They are both normal distributions. I have limited knowledge of how one tests if two distributions are identical, so I apologize for this vague statement. $\endgroup$
    – octopuslegs11
    Commented Jan 6, 2022 at 21:40
  • $\begingroup$ @BruceET Regarding your comment 'once you find the difference in 95th percentiles for two samples, you could simulate to see if they are unusually far apart': how might one go about this? Can I subset on the 95th percentile for each population, then simply visualize the results as a boxplot or ECDF? $\endgroup$
    – octopuslegs11
    Commented Jan 6, 2022 at 21:45
  • $\begingroup$ Not exactly sure I understand what you're proposing. Maybe my answer will persuade you that simulation is not too difficult. Or the note will encourage you to see if your quantiles are nearly normal. Maybe so with samples of size 400,000 and you use 95th percentiles Otherwise @whuber has suggested nonparametric tests.. $\endgroup$
    – BruceET
    Commented Jan 6, 2022 at 22:16
  • $\begingroup$ Normal probability plot of 95th percentiles from 100 normal samples of size 400,000 has points very near a straight line. So 95th percentiles of sample of 400,000 may be close to normally distributed. // Even 99th percentiles not bad. $\endgroup$
    – BruceET
    Commented Jan 6, 2022 at 23:09

1 Answer 1

Reset to default
0
$\begingroup$

Two samples of size 1000 from the same normal population.

set.seed(2022)
x1 = rnorm(1000, 50, 7)
x2 = rnorm(1000, 50, 7)
q1 = quantile(x1,.95); q1
     95% 
61.17923 
q2 = quantile(x2,.95); q2
     95% 
61.98871 
dq = abs(diff(c(q1,q2)));  dq
      95% 
0.8094812

Is this an unusually large discrepancy?

set.seed(106)
m = 10^5;  dq.95 = numeric(m)
for (i in 1:m) {
 x1 = rnorm(1000, 50, 7)
 x2 = rnorm(1000, 50, 7)
 q1 =quantile(x1,.95)
 q2 =quantile(x2,.95)
 dq.95[i] = abs( diff(c(q1,q2) ))
 }
mean(dq.95 >= dq)
[1] 0.22008

No. Not unusually large; about 22% of such comparisons have 95th percentiles farther apart.

hist(dq.95, prob=T, col = "skyblue2")
abline(v = dq, col="red")

enter image description here

Note: As you might guess from the histogram, the distribution of the 95th percentile of a sufficiently large sample is approximately normal. The variance gets larger for percentiles in the far tails. This CLT for quantiles (except the min and max) a fundamental result in the theory of order statistics. Depending on the circumstances of your project, it might be worth your while to see if your samples are large enough to use this asymptotic result.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Relying on the Normality assumption is contraindicated here, because it's so unlikely this assumption would be reliable out in the tails. Use a nonparametric test. $\endgroup$
    – whuber ♦
    Commented Jan 6, 2022 at 22:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.