12
$\begingroup$

I have a weighted sample, for which I wish to calculate quantiles.1

Ideally, where the weights are equal (whether = 1 or otherwise), the results would be consistent with those of scipy.stats.scoreatpercentile() and R's quantile(...,type=7).

One simple approach would be to "multiply out" the sample using the weights given. That effectively gives a locally "flat" ecdf in the areas of weight > 1, which intuitively seems like the wrong approach when the sample is actually a sub-sampling. In particular, it means that a sample with weights all equal to 1 has different quantiles than one with weights all equal to 2, or 3. (Note, however, that the paper referenced in [1] does appear to use this approach.)

http://en.wikipedia.org/wiki/Percentile#Weighted_percentile gives an alternative formulation for weighted percentile. Its not clear in this formulation whether adjacent samples with identical values should first be combined and their weights summed, and in any case its results do not appear to be consistent with R's default type 7 quantile() in the unweighted/equally weighted case. The wikipedia page on quantiles doesn't mention the weighted case at all.

Is there a weighted generalisation of R's "type 7" quantile function?

[using Python, but just looking for an algorithm, really, so any language will do]

M

[1] Weights are integers; the weights are those of the buffers which are combined in the "collapse" and "output" operations as described in http://infolab.stanford.edu/~manku/papers/98sigmod-quantiles.pdf. Essentially the weighted sample is a sub-sampling of the full unweighted sample, with each element x(i) in the sub-sample representing weight(i) elements in the full sample.

$\endgroup$
1

1 Answer 1

5
$\begingroup$

This is one possible approach:

Let's suppose you have a ordered sample $X_1 \le X_2 \le \cdots \le X_n$ with respective weights $W_1, W_2, \ldots, W_n$.

Define $$S_k = (k-1) W_k+ (N-1) \sum_{i=1}^{k-1} W_i$$ so $S_1=0$ and $S_n = (N-1) \sum_{i=1}^{N} W_i$.

For an interpolation of quantile $p$, find $k$ such that $\frac{S_k}{S_n} \le p \le \frac{S_{k+1}}{S_n}$. Your estimate could then be

$$X_k + (X_{k+1}-X_k)\frac{pS_n-S_k}{S_{k+1}-S_k}.$$

I think you will find that if the $W_i$ are all equal then this reproduces R-7. There are other approaches which do too, but I suspect they do not treat all the ordered weights as being equally important.

$\endgroup$
3
  • 2
    $\begingroup$ There may an issue if two values in the sample are equal but have different weights - I have not though about it. $\endgroup$
    – Henry
    Jul 19, 2011 at 0:10
  • $\begingroup$ I've been working on this formula as well. Can I ask where this formula came from? I find it hard to locate the origin of it or any other r type formulas. $\endgroup$
    – Bert Chen
    Jul 20, 2020 at 7:50
  • $\begingroup$ If two values are equal but have different weights, the answer may well be different if you think they are truly the same vs truly different but rounded to the same. If they are truly the same, it's like seeing one value with a higher weight; if they are truly different, the interpolation should be affected. $\endgroup$ Jul 20, 2020 at 8:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.