How is that possible, that in R, SAS or other packages there are so many ways to calculate quantiles? Isn't median just median?

Question

As in the topic. I always thought, that we have a clear, well established definition of a quantile over a vector of numbers. For example - median is such observation, that splits the data set in so, that 50% of data are >= than it and 50% are <= than it. In case of even number of numbers, we take the average of the two consecutive mid-values. And that's clear. Same can be said about other quantiles, as needed, if we only change the fractions.

Then I read, that quantiles - quartiles, percentiles, deciles and all other "-iles" are inferred from the cumulative distribution function, but - at the end of the day - it leads to the same outcomes.

Then I started learning and used two software to practise, SAS, R and SQL. When I compared quartiles calculated by the three tools, I got different results for the median! I read the documentation and found, that there are lots of ways to calculate the quantiles. When I set appropriate option in R or SAS, the discrepancies disappeared, which is fine, but still my concerns didn't disappear.

Isn't median just median? If we have the clear definition taught in school and textbooks, why do we have to care on the right type of quantiles calculated?

And which is better? Is there anything behind the choice? I assume statisticians making SAS and R are well educated people in statistics, so they know what they do. And yet - they chose different algorithms, so even the professionals aren't consistent in this matter.

I heard something that this returns an estimator of the quantile, but if I have my entire data set, the population per se, I don't haver to estimate anything! Which one should I choose then?

Please enlighten me, why the "classic median" calculated by hand hardly matches the results by professional statistical software? Of course, the differences are minor, like 0.5, but they do exist.

If I am asked by my Employer and Clients why I get different medians depending on software, rather than just using the method taught at school, I will have to justify it somehow...

Answer 1

As in so many cases, the simple explanation from school is simple, but not actually true.

The problem is that the CDF doesn't define the median uniquely when the number of observation isn't even (and even less often for other quantiles), and there's more than one other property you might want to use as a tie-breaker. If we have an even number of observations, let's call the interval containing the middle two observations the 'median interval'

The usual mathematical definition is based on the inverse of the CDF, which gives the 'left median', the lower end of the median interval. This also has the valuable property that the median is one of the observed data values -- in the rare cases where you aren't interested in the median as an estimator, but just as a summary of all the data, it should certainly be a data value.

The problem with the left median is symmetry. Why not the right median? The inverse CDF gives you the left median because CDFs are right-continuous, but they could have been left-continuous; mathematicians just picked one possibility. So, the right median is another possibility. Or, you might want some compromise between the two, and that's one way to get the school median. But we've now got at least two answers and possibly three (if you count the right median).

Once we've established that there isn't going to be a single answer, people are going to try to find an 'optimal' answer, for different definitions of 'optimal'. That's where all the other answers come from.

So, why hasn't someone forced everyone to standardise on a particular choice? Well, there has been standardisation, it just has been different in different fields. There's a lot of information on the R help page for quantile and the paper by Hyndman and Fan it references (below) is also helpful.

Since there's no shortage of much more interesting and important battles to fight in statistical practice, very few people have cared about trying to enforce a unique worldwide definition of the median. Well-designed software will let you compute whichever one you want, and that's all you actually need.

Hyndman, R. J., & Yanan, F. (1996). Sample Quantiles in Statistical Packages. The American Statistician, 50(4), 361–365.