25
$\begingroup$

Are there 99 percentiles, or 100 percentiles? And are they groups of numbers, or divider lines, or pointers to individual numbers?

I suppose the same question would apply for quartiles or any quantile.

I have read that the index of a number at a particular percentile(p), given n items, is i = (p / 100) * n

That suggests to me that there are 100 percentiles.. because supposing you have 100 numbers(i=1 to i=100), then each would have an index(1 to 100).

If you had 200 numbers, there'd be 100 percentiles, but would each refer to a group of two numbers. Or 100 dividers excluding either the far left or far right divider 'cos otherwise you'd get 101 dividers. Or pointers to individual numbers so the first percentile would refer to the second number, (1/100)*200=2 And the hundredth percentile would refer to the 200th number (100/100)*200=200

I have sometimes heard of there being 99 percentiles though..

Google shows the oxford dictionary that says of percentile- "each of the 100 equal groups into which a population can be divided according to the distribution of values of a particular variable." and "each of the 99 intermediate values of a random variable which divide a frequency distribution into 100 such groups."

Wikipedia says "the 20th percentile is the value below which 20% of the observations may be found" But does it actually mean "the value below or equal to which, 20% of the observations may be found" i.e. "the value for which 20% of the values are <= to it". If it were just < and not <=, then By that reasoning, the 100th percentile would be the value below which 100% of the values may be found. I have heard that as an argument that there can be no 100th percentile, because you can't have a number where there are 100% of the numbers below it. But I think maybe that argument that you can't have a 100th percentile is incorrect and is based an error that the definition of a percentile involves <= not <. (or >= not >). So the hundredth percentile would be the final number and would be >= 100% of the numbers.

New contributor
barlop is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$
  • 4
    $\begingroup$ I think it unlikely 100 would be a reasonable answer due to its asymmetric treatment of the extremes. Cases can be made for either 99 (as in the definition you quote) or 101. $\endgroup$ – whuber Oct 7 at 18:50
  • 4
    $\begingroup$ Historically quantiles — as we now say generically — were first summary points, and then by extension the bins, classes or intervals they delimit. So three quartiles, including the median, define four bins, and so forth. $\endgroup$ – Nick Cox Oct 7 at 19:40
  • 1
    $\begingroup$ @whuber You write "I think it unlikely 100 would be a reasonable answer due to its asymmetric treatment of the extremes." <-- can you elaborate on that? $\endgroup$ – barlop Oct 7 at 21:47
  • 3
    $\begingroup$ I list early uses of various quantile terms at stats.stackexchange.com/questions/235330/…. If you look within the OED or jstor you will get examples of historical usage. $\endgroup$ – Nick Cox Oct 7 at 23:45
  • 2
    $\begingroup$ @whuber Yes, it appears that what I am referencing is properly called "percentile rank", used in test-score reports & c.: en.wikipedia.org/wiki/Percentile, en.wikipedia.org/wiki/Percentile_rank, ncme.org/resources/glossary. Apologies for adding to confusion. In my defense, the difference appears to hinge on usage of the prepositions "at" vs. "in" (see 1st link). $\endgroup$ – Jeff Y Oct 8 at 15:46
31
$\begingroup$

Both of these senses of percentile, quartile, and so on are in widespread use. It’s easiest to illustrate the difference with quartiles:

  1. the “divider” sense — there are 3 quartiles, which are the values dividing the distribution (or sample) into 4 equal parts:

       1   2   3
    ---|---|---|---
    

    (Sometimes this is used with max and min values included, so there are 5 quartiles numbered 0–4; note this doesn’t conflict with the numbering above, it just extends it.)

  2. the “bin” sense: there are 4 quartiles, the subsets into which those 3 values divide the distribution (or sample)

     1   2   3   4
    ---|---|---|---
    

Neither usage can reasonably be called “wrong”: both are used by many experienced practitioners, and both appear in plenty of authoritative sources (textbooks, technical dictionaries, and the like).

With quartiles, the sense being used is usually clear from context: speaking of a value in the third quartile can only be the “bin” sense, while speaking of all values below the third quartile most likely means the “divider” sense. With percentiles, the distinction is more often unclear, but it’s also not so significant for most purposes, since 1% of a distribution is so small — a narrow strip is approximately a line. Speaking of everyone above the 80th percentile might mean the top 20% or the top 19%, but in an informal context that’s not a major difference, and in rigorous work, the meaning needed should be presumably clarified by the rest of the context.

(Parts of this answer are adapted from https://math.stackexchange.com/questions/1419609/are-there-3-or-4-quartiles-99-or-100-percentiles, which also gives quotations + references.)

New contributor
PLL is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$
  • 2
    $\begingroup$ (+1) This late answer nicely gets to the heart of the matter. $\endgroup$ – Nick Cox Oct 9 at 8:55
  • $\begingroup$ what about en.wikipedia.org/wiki/Percentile says "every score is in the 100th percentile" <-- that sounds like a bin the size of the entire data set, whereas your bins are all equal sizes $\endgroup$ – barlop Oct 10 at 2:49
  • 1
    $\begingroup$ The Wikipedia entry does say that. I can't think of a defence for such wording. Wikipedia is wonderful, except when it is misleading or wrong. That will sound flippant, but all that I can do is encourage anyone watching who is active on Wikipedia to improve the entry. Everyone has to have rules for what they do and don't do, and being active here and in a few other places is my personal limit. $\endgroup$ – Nick Cox Oct 10 at 10:22
5
$\begingroup$

Take this answer with a grain of salt -- it started out fairly wrong and I am still deciding what to do with it.

The question is partly about language and usage, whereas this answer focuses on mathematics. I hope that the mathematics will provide a framework for understanding different usages.

One nice way to treat this is to start with simple math and work backwards to the more complicated case of real data. Let's start with PDF's, CDF's, and inverse CDF's (also known as quantile functions). The $x$th quantile of a distribution with pdf $f$ and cdf $F$ is $F^{-1}(x)$. Suppose the $z$th percentile is $F^{-1}(z/100)$. This provides a way to pin down the ambiguity you identify: we can look at situations where $F$ is 1) not invertible, 2) only invertible on a certain domain, or 3) invertible but its inverse never attains certain values.

Example of 1): I'll leave this for last; keep reading.

Example of 2): For a uniform 0,1 distribution, the CDF is invertible when restricted to [0, 1], so the 100th and 0th percentiles could be defined as $F^{-1}(1)$ and $F^{-1}(0)$ given that caveat. Otherwise, they are ill-defined since $F(-0.5)$ (for example) is also 0.

Another example of 2): For a uniform distribution on the two disjoint intervals from 0 to 1 and 2 to 3, the CDF looks like this.

enter image description here

Most quantiles of this distribution exist and are unique, but the median (50th percentile) is inherently ambiguous. In R, they go half-way: quantile(c(runif(100), runif(100) + 2), 0.5) returns about 1.5.

Example of 3): For a normal distribution, the 100th and 0th percentiles do not exist (or they "are" $\pm \infty$). This is because the normal CDF never attains 0 or 1.

Discussion of 1): For "nice" cdf's, such as with non-extreme quantiles or continuous distributions, the percentiles exist and are unique. But for a discrete distribution such as the Poisson distribution, my definition is ambiguous because for most $z/100$, there is no $y$ with $F(y) = z/100$. For a Poisson distribution with expectation 1, the CDF looks like this.

enter image description here

For the 60th percentile, R returns 1 (quantile(c(rpois(lambda = 1, n = 1000) ), 0.60)). For the 65th percentile, R also returns 1. You can think of this as drawing 100 observations, ranking them low to high, and returning the 60th or 65th item. If you do this, you will most often get 1.

When it comes to real data, all distributions are discrete. (The empirical CDF of runif(100) or np.random.random(100) has 100 increments clustered around 0.5.) But, rather than treating them as discrete, R's quantile function seems to treat them as samples from continuous distributions. For example, the median (the 50th percentile or 0.5 quantile) of the sample 3,4, 5, 6, 7, 8 is given as 5.5. If you draw 2n samples from a unif(3,8) distribution and take any number between the nth and (n+1)th sample, you will converge on 5.5 as n increases.

It's interesting to also consider the discrete uniform distribution with equal probability of hitting 3,4,5,6,7,8. (A die roll plus two.) If you take the sample-and-rank approach outlined above for the Poisson distribution, you will usually get 5 or 6. As the samples get bigger, the distribution for the number halfway up will converge on half fives and half sixes. 5.5 seems like a reasonable compromise here too.

$\endgroup$
  • 2
    $\begingroup$ Your first paragraph has some incorrect information: $F^{-1}$ is indeed unique in many cases, including for the uniform distribution on $[0,1]$ (when $F$ is restricted to $[0,1]$ itself). This has little to do with $F$ being "constant." I think you are making misleading arguments that mix up the roles of continuity, invertibility, and boundedness of support of distributions. Introducing estimators and referring to them also as "quantiles" is interesting but threatens to make things even more confusing. $\endgroup$ – whuber Oct 7 at 19:46
  • $\begingroup$ Good point. I have tried to separate out some cases to clarify that. How would you improve the discussion of continuity? The interpretation of quantiles as estimators is the central point of my answer; they don't really make sense to me without that. $\endgroup$ – eric_kernfeld Oct 8 at 13:08
  • $\begingroup$ Re the latter: quantiles don't need to estimate anything. They are useful in their own right for describing and visualizing data (and often are used only as descriptive statistics). Re continuity: I think most authorities would say that all percentiles exist for discrete distributions. Insisting otherwise is an unnecessary complication. It would also render the results of most software calculations utterly mysterious, which happily provide all quantiles from 0 through 1 (inclusive) for any dataset. In R, for instance, type quantile(0). $\endgroup$ – whuber Oct 8 at 13:12
  • $\begingroup$ This discussion has made me realize that I do not understand quantiles of discrete distributions. I think I should delete this answer. $\endgroup$ – eric_kernfeld Oct 8 at 13:30
  • 1
    $\begingroup$ People vary about this, Eric. When my answers are so wrong as to be misleading, I first delete them. If I see some potential value in part of the answer I edit it to remove (or explain) the misleading part and then undelete it. Others just let things stand and take their lumps in the voting; others add an edit suggesting there may be value in readers seeing where some misunderstanding might have occurred; yet others just delete. You can even completely change the answer if you like, as is sometimes done. $\endgroup$ – whuber Oct 8 at 13:55
2
$\begingroup$

I was taught that an observation in the nth percentile was greater than n% of observations in the dataset under consideration. Which to me implies that there is no 0th or 100th percentile. No observation can be greater than 100% of observations because it forms part of that 100% (and a similar logic applies in the case of 0).

Edit: For what it's worth, this is also consistent with non-academic usage of the term that I've encountered: "X is in the nth percentile" implies that the percentile is the group, not a boundary.

I unfortunately have no source for this that I can point you to.

$\endgroup$
  • 5
    $\begingroup$ Do you have an authoritative reference for what you remember being taught? Note that you are implicitly adopting a definition of "percentile" as being a group of numbers. The other definition quoted in the question is that the percentile is a boundary between such groups. $\endgroup$ – whuber Oct 7 at 18:51
  • 1
    $\begingroup$ That doesn't make sense to me because suppose your data is 2,2,2,2,2,2,2,2,2,2,2 so an item in one quantile is equal to an item to its left in a prior quantile. So an item in the nth quantile is not greater than all quantiles left of it. So an item in the nth percentile is not greater than n% of observations in the dataset. It's >= n% of observations in the dataset, but not simply >. And hence you can have a 100th pecentile.. what do you make of that logic? $\endgroup$ – barlop Oct 7 at 21:45
  • 3
    $\begingroup$ Many definitions come under strain if all values are identical! $\endgroup$ – Nick Cox Oct 7 at 23:46
  • 2
    $\begingroup$ Those of mathematical bent abstract and idealise while those who write software need to deal with the messiness of data. Your example of 16 values would be treated differently by software I know which follows a rule that identical values must be binned identically (and I agree). I am surprised that you did not agonise over data with 15 or 17 values where even if all values are distinct no rule can divide data into 4 bins of equal size. $\endgroup$ – Nick Cox Oct 8 at 7:49
  • 3
    $\begingroup$ What's the similar logic for zero? Doesn't "greater than zero percent of the observations" mean "equal to or smaller than all the observations", i.e. the 0th percentile would be the lowest observed value? $\endgroup$ – ilkkachu Oct 8 at 12:35
2
$\begingroup$

There are other ways to calculate percentiles, what follows, is not the only one. Taken from this Source.


The meaning of percentile can be captured by stating that the $p$th percentile of a distribution is a number such that approximately $p$ percent ($p\%$) of the values in the distribution are equal to or less than that number. So, if $28$ is the $80$th percentile of a larger batch of numbers, $80$% of those numbers are less than or equal to $28$.

To calculate percentiles, sort the data so that $x_1$ is the smallest value, and $x_n$ is the largest,

with $n$ = total number of observations, $x_i$ is the $p_i$th percentile of the data set where:

$p_i = \dfrac{100(i - 0.5)}{n}$

Example from the same notes for illustration:

enter image description here

To take a single example, $7$ is the $50$th percentile of the distribution, and about half of the values in the distribution are equal to or less than $7$.

If you had 200 numbers, there'd be 100 percentiles, but would each refer to a group of two numbers.

No.

Assuming the numbers are sorted in ascending order moving from $x_1$ to $x_\mathrm{200}$. In this case the percentiles are:

$\dfrac{100(1-0.5)}{200}$, $\dfrac{100(2-0.5)}{200}$, $\dfrac{100(3-0.5)}{200}$, $...$

resulting in

$0.25, 0.75, 1.25 ... $ percentiles corresponding to indices $1, 2, 3, ...$

$\endgroup$
  • 3
    $\begingroup$ The first sentence looks great, and one of the most important words is approximately, Thereafter this is a careful explanation of just one recipe. What's key is that there are several recipes and most if not all have some defensible logic about them (sometimes the logic is to keep things as simple as possible). See the Hyndman and Fan paper referred to in many threads here on CV. I doubt that many people would take your last paragraph as the way to report percentiles for your example. $\endgroup$ – Nick Cox Oct 8 at 9:25
  • $\begingroup$ @Nick Cox Thank you for the insightful comment. About the last paragraph I believe the method should work fine when all the observations are different from each other. In case of repeated numbers there will not be unique percentile for the same number which doesn't sound good. Could you kindly suggest how to deal with the case. And could you also point out the potential pitfalls in the last paragraph. $\endgroup$ – naive Oct 8 at 9:49
  • 1
    $\begingroup$ I don't think I want or need to add to what is already well explained in journal literature. First, you have some favourite software for this. See what it documents and what it does. Second, I've not calculated percentiles by hand for some decades, and none of us needs to. Third, my point about the last para: I guess no-one wants to be told that the observed data points are the 0.25, 0.75, 1.25, ... percentiles. What people do want varies, but in my experience it's most commonly wanting summaries such as 1, 5, 10, 25, 50, 75, 90, 95, 99% points as well as the sample extremes. $\endgroup$ – Nick Cox Oct 8 at 9:57
  • 1
    $\begingroup$ I've just noticed that you assert that 0.5 is in EDA jargon often called the p-value for the median. Not in my reading, and even if you can find examples that is terrible terminology given an overwhelming majority sense for p-value as observed significance level. $\endgroup$ – Nick Cox Oct 8 at 9:59
  • $\begingroup$ I will go through the paper that you suggested. Thank you $\endgroup$ – naive Oct 8 at 9:59
0
$\begingroup$

Note- I will accept somebody else's answer rather than mine. But I do see some useful comments so I'm just writing an answer that mentions those.

Based on Nick's answer "-iles" terminology for the top half a percent

it seems that the terms are ambiguous, and I suppose (based on my understanding of that post), better terminology would be X% point, and X%-Y% group; so quantile point(so for quartile points that could be anything from 0 to 4); quantile group ranging from X quantile point to Y quantile point.

Either way one would get 101 for percentiles, although one comment suggests that one could refer to 101 points (I suppose if you counted percentile points, and only integers), but even then, if one speaks of 1st, 2nd, 3rd, percentile or quantile, it's counting and one can't count the first as 0, and you can't have e.g. more than 4 quartiles or more than 100 percentiles. So if talking 1st, 2nd, 3rd, that terminology can't really refer to point 0. If somebody said 0th point, then while it's clear they mean point 0, I think they should really say quantile point 0. Or Quantile group at point 0. Even computer scientists wouldn't say 0th; even they count the first item as 1, and if they call it item 0, that's an indexing from 0, not a count.

A comment mentions "There can't be 100. Either 99 or 101, depending on whether you count maximum and minimum". I think there's a case for 99 or 101, when talking about quantile points rather than groups, though I wouldn't say 0th. For n items, An index may go from 0...n-1 and one wouldn't write th/st e.g. 1st, 2nd etc, on an index(unless perhaps the index happened to index the first item as 1). But an index starting the first item with index of 0 isn't a 1st, 2nd 3rd count. e.g. item with index of 0 is the 1st item, one wouldn't say 0th and label the second item 1st.

New contributor
barlop is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$
  • $\begingroup$ Any ambiguity was introduced by those who departed from clear historical precedent. It doesn't bite hard in practice. $\endgroup$ – Nick Cox Oct 8 at 8:48
  • $\begingroup$ All mathematicians start counting at zero. The concept is simple and natural: saying the word "zero" out loud announces one's intention to count. Then one makes some (perhaps arbitrary) one-to-one assignment of the sequence of words "one," "two," "three," etc. to the objects being counted. The last of those words (if there is a last) is equated with the cardinality of the set. The beauty of this idea is that when there are no elements in the set, the last word said was "zero," which is the unique correct value. $\endgroup$ – whuber Oct 8 at 11:32
  • $\begingroup$ @whuber you write "All mathematicians start counting at zero" <-- Where do you think I said otherwise? $\endgroup$ – barlop Oct 8 at 12:35
  • $\begingroup$ "it's counting and one can't count the first as 0". $\endgroup$ – whuber Oct 8 at 12:48
  • 1
    $\begingroup$ @whuber possibly many might, I think many years ago I might have, as when studying computer science, I heard sometimes that computer scientists count from 0, unilke mathematicians(that's not your claim or mine) , but after some deep thought I got more clarity and realised that computer scientists and mathematicians both count from 0.. The difference is computer scientists often use an index and the index indexes the first item as 0.(but still count would be 1).. $\endgroup$ – barlop Oct 8 at 13:01

Your Answer

barlop is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.