Correcting for sample size in skewed data Empirically I have noticed that when data is skewed, and when for example interested in the 99th percentile of latencies, that it is difficult to compare experiments with different sample sizes
Imagine I run an experiment for 500 iterations and then afterwards for 2000 iterations, then the p99 when iter=2000 is simply higher
This is example descriptives data:
count    583.0000000000
mean     810.3430531732
std     1472.5106312310
min      112.0000000000
1%       166.6400000000
10%      234.0000000000
25%      274.5000000000
50%      373.0000000000
75%      779.5000000000
90%     1575.0000000000
99%     7404.4400000000
max    16479.0000000000

How can I make this data comparable between experiments correcting for the sample size? Is there some formula?
 A: Trying to reproduce what you have noticed is difficult without knowing the distribution involved, but using a distribution with $F(x)=1-\frac{100}{x}$ and $f(x)=\frac{100}{x^2}$ for $x> 100$ (a Pareto distribution with shape $\alpha=1$ and minimum $100$) seems to give roughly the same sort of shape which is heavily skewed.
The following R code allows simulation of the $99\%$ quantile samples from this
set.seed(2023)
quantile <- 0.99
largersamplesize <- 2000
smallersamplesize <- 500
cases <- 10^4
samplequantile <- function(samplesize, p){
  quantile(100/runif(samplesize), p)
  }
simslarger  <- replicate(cases, samplequantile(largersamplesize,  quantile))
simssmaller <- replicate(cases, samplequantile(smallersamplesize, quantile))
plot(density(simslarger),   col="red")
lines(density(simssmaller), col="blue")
abline(v=100/(1-quantile))

with the following empirical densities (red for the larger sample, blue for the smaller sample)

You can see that the simulated $99\%$ quantiles have wide distributions around the population $99\%$ quantile (the vertical black line), that both distributions are right-skewed, and that the larger sample size leads to greater concentration of the sample quantile.
It is quite possible that the quantile from the smaller sample can be greater than the quantile from the larger sample but, thanks to the skewness, the larger sample quantile is slightly more likely to be larger in a particular case. Using this simulation suggests a probability of about $56.5\%$, though different simulations would produce slightly different estimates.
mean(simslarger > simssmaller) # probability larger sample gives higher value
# 0.565 with this seed

The effect will be smaller and the concentrations narrower with even larger sample sizes or looking at a less extreme quantile.
A: There are some nasty cases to think of: some probability distributions with a fat tail have an infinite mean value. One example is $P(X=x) = \frac{1}{x^2}$.
If you are interested in the tail, then big samples become all the more important.
