# Comparing outliers in two distributions

I apologize in advance as I am not well-versed in statistics, but I hope that this question makes sense.

I have 2 populations which are normally distributed and have a near-identical mean. I would like to know if the upper quantiles (say >0.99) differ - ie, does one population extend further than the other, and how can I see if the difference is significantly different?

One approach I have thought to take is to subset my dataset on just the upper quantiles, then plot the ECDF, but I don't know that this is a correct approach.

• It depends on whether the two populations have the same distribution. Difficult to know what 'nearly identical' might mean. Then the answer depends on sample sizes; as normal samples become larger, there is more chance to see values far into the right tail. The distribution of the max may be hard to handle, so you should not ask for 99th percentile unless you have a very large sample. Maybe 95th of 97th percentiles would be safer. // Once you find the difference in 95th percentiles for two samples, you could simulate to see if they are unusually far apart. Jan 6, 2022 at 21:35
• @BruceET Thanks for the response - I agree that as the sample size becomes larger, there is more chance to see values that extend further into the right tail. I've randomly downsample my dataset so that each population has the same number of events (Approx 400,000 events). I am not sure how to test that the distributions are non-identical. They are both normal distributions. I have limited knowledge of how one tests if two distributions are identical, so I apologize for this vague statement. Jan 6, 2022 at 21:40
• @BruceET Regarding your comment 'once you find the difference in 95th percentiles for two samples, you could simulate to see if they are unusually far apart': how might one go about this? Can I subset on the 95th percentile for each population, then simply visualize the results as a boxplot or ECDF? Jan 6, 2022 at 21:45
• Not exactly sure I understand what you're proposing. Maybe my answer will persuade you that simulation is not too difficult. Or the note will encourage you to see if your quantiles are nearly normal. Maybe so with samples of size 400,000 and you use 95th percentiles Otherwise @whuber has suggested nonparametric tests.. Jan 6, 2022 at 22:16
• Normal probability plot of 95th percentiles from 100 normal samples of size 400,000 has points very near a straight line. So 95th percentiles of sample of 400,000 may be close to normally distributed. // Even 99th percentiles not bad. Jan 6, 2022 at 23:09

Two samples of size 1000 from the same normal population.

set.seed(2022)
x1 = rnorm(1000, 50, 7)
x2 = rnorm(1000, 50, 7)
q1 = quantile(x1,.95); q1
95%
61.17923
q2 = quantile(x2,.95); q2
95%
61.98871
dq = abs(diff(c(q1,q2)));  dq
95%
0.8094812


Is this an unusually large discrepancy?

set.seed(106)
m = 10^5;  dq.95 = numeric(m)
for (i in 1:m) {
x1 = rnorm(1000, 50, 7)
x2 = rnorm(1000, 50, 7)
q1 =quantile(x1,.95)
q2 =quantile(x2,.95)
dq.95[i] = abs( diff(c(q1,q2) ))
}
mean(dq.95 >= dq)
 0.22008


No. Not unusually large; about 22% of such comparisons have 95th percentiles farther apart.

hist(dq.95, prob=T, col = "skyblue2")
abline(v = dq, col="red") Note: As you might guess from the histogram, the distribution of the 95th percentile of a sufficiently large sample is approximately normal. The variance gets larger for percentiles in the far tails. This CLT for quantiles (except the min and max) a fundamental result in the theory of order statistics. Depending on the circumstances of your project, it might be worth your while to see if your samples are large enough to use this asymptotic result.

• Relying on the Normality assumption is contraindicated here, because it's so unlikely this assumption would be reliable out in the tails. Use a nonparametric test.
– whuber
Jan 6, 2022 at 22:11