1
$\begingroup$

From a data set, I know the mean, P10, P50 and P90. How do I estimate the cumulative probability of the mean?

The reason is that my data set may have extreme events, pushing the mean value to a high percentile.

It always hold that P50 < mean < P90. So I would like to have a better estimate of the "percentile" of the mean.For now I'm just using linear interpolation and the result is ~60%. But in this region a linear interpolation will give a lower value.

Improve this question
$\endgroup$
  • $\begingroup$ What does "It always hold that P50 < mean < P90" mean? That it is always true for your kind of data? It is not always true for any data, as Glen_b showed. If it's always true for your data some extra context would be helpful to provide useful answers. $\endgroup$ – Pere Dec 26 '16 at 18:40
  • $\begingroup$ Btw, the question title "estimate percentile of mean from other percentiles" suggest that you see the mean as a percentile. Are you sure that there isn't a confusion of mean with median? Median is actually a percentile, but mean isn't. $\endgroup$ – Pere Dec 26 '16 at 18:43
  • $\begingroup$ I believe the question can be summarized as: $F[x]\equiv\Pr[X<x],\,F[x_{10},x_{50},x_{90}]=[0.1,0.5,0.9],\,x_{50}<\bar{x}<x_{90}\implies{F}[\bar{x}]=?$ $\endgroup$ – GeoMatt22 Dec 26 '16 at 18:55
  • $\begingroup$ that is true for the data in question. let's assume that God has very large data sample, in fact more than 50 years of recordings of weather data every 3 hours. but God is evil and will not give me access to all his data, he'll (or she?) will only allow me to know the 10th, 50th and 90th percentiles and the mean of the sample, which will look something like 5, 9, 18 and 14 respectively. Now, if I had all the data, I would like to know which percentile is closest to the mean. I could go on and start a very long process of requesting access to the godly data (which I'm doing by the way)... $\endgroup$ – Raf Dec 26 '16 at 18:58
  • $\begingroup$ but while that doesn't happen, I'm just after a good enough way to approximate it :) $\endgroup$ – Raf Dec 26 '16 at 18:59
2
$\begingroup$

I believe the question can be summarized as $$F[x]\equiv\Pr[X<x],\,F[x_{10},x_{50},x_{90}]=[0.1,0.5,0.9],\,x_{50}<\bar{x}<x_{90}\implies{F}[\bar{x}]=?$$ As noted by many responses, there is not enough information here to solve the problem uniquely.

However building on the OP suggestion of linear interpolation, a simple approach would be to just extend the order of the polynomial interpolation. So for example we might assume $$F[t]=a+bt+ct^2 \,,\,t=\frac{x-x_{50}}{x_{90}-x_{50}}$$ where the coefficients must satisfy the constraints $F_0=0.5$ and $F_1=0.9$. From the definition of a CDF and PDF, we must also have $F^\prime[t]\geq{0}$.

Together these constraints give $$a=F_0\,,\,c=F_1-F_0-b\,,\,0\leq{b}\leq{2}(F_1-F_0)$$ so we have $$t^2\leq\frac{F[t]-F_0}{F_1-F_0}\leq{t}(2-t)$$

Note that $t\in[0,1]$, and in the linear case the expression in the middle reduces to $t$, so this includes the OP's linear interpolation solution as a special case.

If the OP was intended to be on the "tail", in the sense that $c\leq{0}$ so the PDF is decreasing over the interval, then the lower bound above can be increased from $t^2$ up to $t$ (while the upper bound stays the same, at $t(2-t)$). This is how I interpret the OP comment "in this region a linear interpolation will give a lower value".

Again I will end with the caveat that this is not a general solution, as there is no reason in general that a quadratic approximation will be appropriate for an arbitrary CDF of the $[x_{50},x_{90}]$ interval!

Improve this answer
$\endgroup$
5
$\begingroup$

You can't do this without some strong assumptions about what the extreme tail can do.

You could have two different data sets (or indeed distributions), one where the mean is below the 10th percentile and one where the mean is above the 90th percentile, yet the three percentiles you mention could be the same for both.

For example consider these two samples:

Sample 1:
40  96  97  98  99 100 101 102 103 104 105

Sample 2:
95  96  97  98  99 100 101 102 103 104 160

According to the quantile function in R (with default definition of sample quantiles), both these samples have the same values for the three quantiles you mentioned:

> quantile(sample1,p=c(.1,.5,.9))
10% 50% 90% 
 96 100 104 
> quantile(sample2,p=c(.1,.5,.9))
10% 50% 90% 
 96 100 104

But the mean is below the 10th percentile in the first sample, and above the 90th percentile in the second:

> mean(sample1)
[1] 95
> mean(sample2)
[1] 105

With larger samples you could move the mean deeper, past more tail quantiles; you could put the mean below the first percentile or above the 99th percentile, for example.

Which is to say, if all you know is those three percentiles, you really can't pin the percentile of the mean down much at all -- it must lie above the minimum and below the maximum, but it could be above or below any other observation.

Improve this answer
$\endgroup$
  • $\begingroup$ I think this is the best answer. The terms "cumulative probability of the mean" and "how conservative the mean is related to P(50)" don't make sense to me. Showing that the mean could be anywhere between the minimum and the maximum show how sensitive the mean is to these two values. Your examples show that an extreme minimum pulls the mean toward it and an extreme value for the maximum pulls it toward the maximum. If the OP wants to interpret his own data he should mention what values the mean P10, P50 and P90 are. $\endgroup$ – Michael R. Chernick Dec 24 '16 at 18:40
  • $\begingroup$ Then we can at least roughly know where the mean actually falls and what the empirical cdf is at the mean. That may be what the OP means by "cumulative probability of the mean". We also would know how far above or below P50 the mean is. But which direction does the OP think is conservative. Note that all the quantities we are talking about are sample values and not population parameters. $\endgroup$ – Michael R. Chernick Dec 24 '16 at 18:47
  • $\begingroup$ please see updated question. the mean is never below P50 and never above P90. also note by "estimate" all I want is a rough approximation (can't really expect more than that with 4 points). $\endgroup$ – Raf Dec 26 '16 at 18:22
  • $\begingroup$ @Michael Even though the quantiles are sample values the same issues we see in samples hold for distributions. $\endgroup$ – Glen_b Dec 26 '16 at 18:48
  • $\begingroup$ @Raf nothing about your modifications alters the facts here -- if you only specify three those three quantiles and nothing more then the mean can be below the 10th percentile or above the 90th (or indeed outside any other quantile you like in a large enough sample). If you add an assumption that the mean exceeds the 50th percentile then this only adds exactly that restriction, so the mean could be anywhere between the 50th percentile to above the 90th percentile. It's not possible to say "roughly" where it is without assumptions about the shape (especially the behavior of the extreme tail) $\endgroup$ – Glen_b Dec 26 '16 at 18:54
1
$\begingroup$

I am not aware of a function that does it. However you should be able to simply count. Some (python'ish) pseudo code:

len(data[data.value <= data.value.mean()]) / len(data.value)
Improve this answer
$\endgroup$
  • 1
    $\begingroup$ In words, the percentile you seek (really, not the percentile at all, but the corresponding cumulative probability) is roughly the fraction of values that are less than or equal to the mean, which requires access to the original data vector. $\endgroup$ – Nick Cox Dec 23 '16 at 11:05
-1
$\begingroup$

Think I have the answer, from the definition of the mean using probabilities:

$$\overline x = \sum x_i p_i$$

Now I have mean, P10, P50 and P90:

$\overline x, x_{p10}, x_{p50}, x_{p90}$.

They are respectively associated to the cumulative probabilities:

$q_{\overline x}, 0.1, 0.5, 0.9$.

And the best we can assume with this small set is that their probabilities are respectively (I know from my data that mean is larger than P50, or $\overline x > x_{p50} \to q_{\overline x} > 0.5$):

$q_{\overline x}-0.5, 0.1, 0.4, 0.9-q_{\overline x}$

So:

$$\overline x = 0.1 x_{p10} + 0.4 x_{p50} + (q_{\overline x} - 0.5) \overline x + (0.9 - q_{\overline x})x_{p90}$$ $$\cdots$$ $$q_{\overline x}=\frac{0.1 x_{p10} + 0.4 x_{p50}+0.9 x_{p90}-1.5 \overline x}{x_{p90}-\overline x}$$

Improve this answer
$\endgroup$
  • 1
    $\begingroup$ I assume that P10, P50 and P90 are percentiles. Therefore, I'm afraid that in $\overline x = 0.1 x_{p10} + 0.4 x_{p50} + (q_{\overline x} - 0.5) \overline x + (0.9 - q_{\overline x})x_{p90}$ you are assuming that valule $x_{p10}$ has a 10% probability (and the same for the other values), but the truth is that 10% probability goes to all values lower or equal than $x_{p10}$, not just to $x_{p10}$. Furthermore, probabilities in the formula sum up to 0.9 instead of 1. Am I missing something? $\endgroup$ – Pere Dec 23 '16 at 11:42
  • $\begingroup$ see this stats.stackexchange.com/a/104620/96344 $\endgroup$ – Raf Dec 26 '16 at 18:15
  • $\begingroup$ That answer is wrong because it's estimate is biased: it assumes that all points of a given interval are in the maximum of that interval. The error made this way is bigger when intervals are bigger, and yours are very big. Furthermore, they made a somehow arbitrary correction by doubling the weight of first interval, but here you don't even do that correction. $\endgroup$ – Pere Dec 26 '16 at 18:23
  • $\begingroup$ estimate = approximate calculation. Indeed the intervals are large, but that is what I have to work with since I can't have more data. I don't think any other method would give a more precise answer given these restrictions. $\endgroup$ – Raf Dec 26 '16 at 18:30
  • $\begingroup$ When intervals are bounded, a less biased approximation is to suppose the points are in the centre of the interval, not in one of its ends. Furthermore, you need to account for all the points, and probabilities (coefficients) for computing the mean must sum 1. Yours sum 0.9. $\endgroup$ – Pere Dec 26 '16 at 18:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.