We're working with latency data and people are concerned with the worst case latencies. They're using the 99th and 99.9th percentile as KPIs (key performance indicators). We know that an unbiased estimator of even the median doesn't exist: An unbiased estimate of the median. And the bias issue becomes much worse for the high percentiles than it is for the median. In contrast, the sample mean is an unbiased estimator. To an extent, the sample mean will catch changes in the tail as well. I was wondering if there are any alternate KPI's that do better on the bias and yet are also focused on the tail. The conditional sample mean (conditional on the samples being larger than some number) was one measure that came to mind. But then the question is, what should the conditional threshold be? Should it be some static number or again a percentile? Any other descriptive statistics that focus on the tail and yet do better on the bias issue?
$\begingroup$
$\endgroup$
3
-
3$\begingroup$ You mention conditional tail expectation (it has a variety of other names) -- you're after the right tail rather than the left tail as treated at the link, but this is common in some application areas. Choice of where to place the threshold is very much problem dependent (we can't tell you what matters for your circumstances -- an alternative might be to consider a sequence of such values). There's lots of papers and books that treat these measures. $\endgroup$– Glen_bCommented Oct 4, 2021 at 22:08
-
$\begingroup$ Can you please spell out KPI? $\endgroup$– kjetil b halvorsen ♦Commented Oct 5, 2021 at 0:35
-
$\begingroup$ Key performance indicator. Basically, sample statistics that are indicators for performance. $\endgroup$– ryu576Commented Oct 5, 2021 at 17:48
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Probably this falls into topic of risk measures as mentioned by Glen_b. Expected Shortfall could be used. But it isn't a coherent risk measure. Entropic value at risk (EVaR) would be an option for coherent measure, if coherency axiom's are valid/important for latency measurement.
answered Oct 5, 2021 at 0:15