Should 99 percentile of a union of two sets be the average of 99 percentile of each set?

Question

For example, the 99 percentile value of list A is p99_a, the 99 percentile value of list B is p99_b, list C is the full set of A and B, should the 99 percentile value of list C be the 99 percentile value of p99_a and p99_b or the average value of p99_a and p99_b?

I always thought it should be the former one, however, I tried it on codes:

import numpy as np
import random
data = []
p99list = []
for i in range(10000):
    one_data = [random.randrange(10000) for x in range(1000)]
    data += one_data
    p99list.append(np.percentile(one_data, 99))

print('p99 of all data: \t' + str(np.percentile(data, 99)))
print('average of p99: \t' + str(np.average(p99list)))
print('p99 of p99 : \t' + str(np.percentile(p99list, 99)))

The results were:

p99 of all data:    9899.0
average of p99:     9889.646635999998
p99 of p99 :    9952.01

It showed that average of p99 was closer to the p99 of all data than p99 of p99. On the Contrary, if I changed the sixth line of code to as follows (on the purpose of simulating the response time of HTTP requests from one server):

one_data = [random.uniform(0.2, 0.4) for x in range(1000), random.uniform(1.0, 1.2) for y in range(5)]

I ran the code again, and the results were:

p99 of all data:    0.39801099789433964
average of p99:     0.37998116766051837
p99 of p99 :    0.39904330107367425

It turned out that p99 of p99 was closer to the p99 of all data than average of p99.

So which one is more accurate?

Answer 1

You can't say in general.

Suppose $A=\{1, \dots, 100\}$, so its 99th percentile is $q^A_{99}=99$, and $B=\{101, \dots, 200\}$, so its 99th percentile is $q^B_{99}=199$. Then the 99th percentile of $C:=A\cup B$ is $q^C_{99}=198$, which is about as far away from the mean of $q^A_{99}$ and $q^B_{99}$ (which is $149$) as it can get.

Quantiles of a set are bounded by quantiles of subsets, so all you know is that $$\min\{q^A_{99},q^B_{99}\}\leq q^{A\cup B}_{99}\leq \max\{q^A_{99},q^B_{99}\}.$$ There is no general way to improve this much.

Answer 2

(Copying my answer from Stack Overflow)

The 99 percentile of a union of two sets (even if they are of equal size) cannot be simply deduced from the 99 percentiles of both sets.

For example:

say that the first set has only 0's, and the second set has only 1's. Therefore, the 99 percentile of the first set is 0, and the 99 percentile of the second set is 1. But in that case, the 99 percentile of their union is 1 - not related to the average of 0 and 1. (Of course, in this example, it is their maximum - but it's not hard to construct a counterexample for that too...)

There might be something to say if both sets come from a normal distribution, but if you're looking at the 99 percentiles of some real-world data, then the 99 percentile usually represents outliers that do not have a clean normal distribution.