3
$\begingroup$

I'm not sure that variance is technically the right term, so allow me to explain... I'm measuring two data sets at work, looking at the 50th, 90th, and 99th percentile for each data set on a monthly basis. Data set A generally has about 10x as many observations as data set B.

If any of these metrics changes enough in one month for either data set, my bosses want an explanation for why it changed. The threshold they've historically used for whether an explanation is necessary is a 3% delta month-over-month. (For various long reasons, doing an actual statistical significance test is not possible for our circumstances.) This 3% threshold was established when we were only looking at data set A, where anecdotally it seems to be about right. But is now also being applied to data set B. Obviously since data set B has one tenth as many observations, it breaches this threshold much more frequently.

I want to make the case to say something like "data set B has one tenth as many observations, so we should allow for a threshold that is X times larger". I'm just not sure how to determine what X should be. I have this vague intuitive sense from old stats classes that it should be proportional to sqrt(N), i.e. if data set A is allowed to vary up to 3% from month to month, then data set B should be allowed to vary by up to 3% * sqrt(10) =~ 9.5% per month.

My question is: Am I right? And regardless of whether I'm right, what's the actual statistical argument for whatever the right answer is? How much should data set B be allowed to vary assuming that we allow data set A to vary by 3%?

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ Depending on how your data are distributed, the 50% quantile could be much more or much less variable than the 99% quantile, so the threshold might need to be different for each quantile in addition to depending on the sample size. $\endgroup$
    – Stephan Kolassa
    Commented Jun 13, 2013 at 7:42
  • $\begingroup$ Yes, there are many problems with the way we do this, and what you pointed out is one of them. I'm trying to start to bring some statistical rigor to the process; this is just the first step. For now I just know enough to say "we are definitely doing this wrong, but I'm not sure yet how to do it right". $\endgroup$
    – FuriousGeorge
    Commented Jun 13, 2013 at 15:50
  • $\begingroup$ To Kjetil: We are measuring the latency of a webpage. We have one sample for each visit. n is about 100M for data set B, so even the 99th percentile is looking at about 1M samples. Based on that it sounds like it's fair to approximate it at roughly sqrt(n). $\endgroup$
    – FuriousGeorge
    Commented Jun 13, 2013 at 15:56

1 Answer 1

Reset to default
1
$\begingroup$

We really need to know more about your situation! Are your data an iid sample, a time series, or some other? Comments below assume the iid situation. The 50% quantile (the median) have variance scaling as the mean, that is, with $\sqrt(n)$. The 99% quantile can be very different, dependent on the sample size. If the sample size is really huge (many thousands), then it will scale likewise. For small $n$, say $N=100$, that quantile is an extreme, and we need extreme order statistics theory to approximate its distribution.

Distribution theory for extreme order statistics depends on the form of the tail of the distribution, so you will need to study that. Specifically, it depends on the so-called tail coefficient, so you need to estimate that. If you can give more information, maybe we can help more.

Improve this answer
$\endgroup$
2
  • $\begingroup$ We are measuring the latency of a webpage. We have one sample for each visit. n is about 100M for data set B, so even the 99th percentile is looking at about 1M samples. Based on that it sounds like it's fair to approximate it at roughly sqrt(n). $\endgroup$
    – FuriousGeorge
    Commented Jun 13, 2013 at 15:59
  • $\begingroup$ You should be right in that conclusion! $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jun 13, 2013 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.