Averaging rankings of small geographical areas to find an average ranking for a larger area I have a friend who is very intelligent with a professional background in accountancy but no formal mathematical or scientific training. He is studying for a PhD (as a mature student) looking at how the philosophy of accounting affects big science projects
We have been talking about statistics. As a personal piece of work, he has been using a database of national indices of mass deprivation to show how different areas are getting more or less deprived over time. He has been grouping smaller areas together to create bigger areas. All fine so far. However, he has been averaging the rankings of the smaller areas to get an average ranking for the larger area and then comparing how the larger area's ranking varies over time
This just seems wrong to me - the ranking has no direct link to a measurable value, and averaging ranks "works arithmetically", but to me doesn't seem to provide a meaningful stat. It seems obvious (dread word!) that you need to work out the values of deprivation for the larger areas and then rank them all again (as long as all larger areas are approximately equivalent)
As I said, he is very intelligent and an excellent debater. His view is that statistics is only a representation of the truth, and as such he is able to select how to represent this truth
Who is right? If I'm right, what arguments could I use to help him understand?
 A: This is one of those areas with few, clearcut right or wrong answers, but some ways are clearly better than others.
All rankings are fraught with many, many pitfalls as this article by Malcolm Gladwell about them makes clear. Your friend's problems begin with his assumption that the different indices of "mass deprivation" can be summarized into a single number. Among the problems Gladwell enumerates are:
1) Omitted variables bias, it can be really hard to identify the right set of measures for ranking. If there are key constructs that define a construct that are left out of an analysis, then any resulting ranking will be compromised
2) The subjective nature of the weightings employed
3) The quality of the input information is often suspect 
4) The nearly infinite ways to develop the rankings
5) The loss of complex information in generating an ordinal ranking
http://www.newyorker.com/magazine/2011/02/14/the-order-of-things
Simplistic and easily derived methods like your friend's will pass muster among the technically illiterate and naive but are guaranteed to flame out if anyone with any statistical insight takes a look under the hood -- like a PhD dissertation committee. In other words, there is a serious element of risk in what he's proposing. 
At the end of the day, there is no substitute for analytic rigor and stubbornness is no answer for ignorance.
A: The best illustration for why something problematic might be going on when averaging rankings is probably an example of the following ranking of areas (as an example I will use average income and assume that income per head is a relevant number):


*

*Large area 1, small geographical sub-area A: population 100, average income 100,000 USD

*Large area 2, small geographical sub-area C: population 10000, average income 10,001 USD

*Large area 2, small geographical sub-area D: population 10000, average income 10,000 USD

*Large area 1, small geographical sub-area B: population 9900, average income 1,000 USD


The average rank for both large areas is 2.5, but the average income per head is much higher in large area 2.
You can probably also construct all sorts of examples with small differences such as {10,001,10,002,10,003 USD} vs. {9,997, 9,998, 9,999 USD} causing major changes in raking and obscuring massive differences in some sub-areas (e.g. if you add another sub-area with average income 100,000 USD to the second group, it still has the lower average rank). A classic example of ranking can be abused is gerrymandering.
Better approaches might include: calculating the metric of interest on a larger area level (if possible based on the available data), using hierarchical models etc.
A: Disclaimer: I'm not a statistician, but I'm going to provide a completely contrived example of why what he is doing might have issues:
Consider the following measures of deprivation for six small areas (lower is more deprived, so worse).
1, 2, 100, 3, 4, 5

It's clear the ranking is:
6, 5,   1, 4, 3, 2

(1 is best).
We are going to group the first three areas in one "super area" and the latter three in another area.
The average ranking for the first "super area" is (6+5+1)/3 = 4. The average ranking for the second area is (4+3+2)/3 = 3. So by his metric, the second area has a better "score" than the first. However, it's clear that this isn't the case: it's not hard to see that the average measure for the first area is (100+2+1)/3 = 34.333... and the average measure for the second area is (3+4+5)/3 = 4, and higher is better. So in fact, his "ranking" is misleading.
This is because rankings abstract the original measure away, so there is no way of knowing just how different the underlying measures are.
Of course, this example is completely contrived, but like Ansombe's Quartet I hope it highlights the issues that could arise when using such an statistic.
Does your friend have access to the underlying statistics?
Finally, aggregating areas is hard, as gerrymandering shows how easily it can be abused. How is he deciding to do it? 
Edit: I suppose that is an example of point 5. of DJohnson's answer.
