Rank correlation and demonstrating reasonable proficiency in rank ordering

Question

Background

I'm working on a new effort estimation model (dissertation work) which involves software engineers organizing tasks (rank ordering) in order of increasing effort based solely on their understanding of the requirements. For example, writing a function to calculate an average should require less effort than writing a function to calculate an integral.

A software engineer's ability to rank order tasks in a reasonably proficient manner is essential to successfully constructing an estimate with my new model. Put another way, a software engineer should be able to get enough of the items in the correct rank order that the estimate becomes useful for project planning purposes. Some errors are expected and can be accounted for by the model (estimation is not an exact science, after all).

I believe I can use one of several rank correlation measurements to compare the software engineer's rankings to the "correct" rankings (as defined by historical data of actual effort).

Question

How can I define reasonably proficient in terms of Spearman's ρ, Kendall's τ, etc.?

One possibility that has occurred to me is "better than random." Assume that anything better than the average of all possible arrangements indicates some level of intelligence in the ranking (ρ > 0 for Spearman, maybe?). This doesn't inspire a lot of confidence, though.

Several references talk about the statistical significance of 10%, 5%, 1%, etc., but give no guidance on why these values are selected. I can't just make up a required level of significance. I do have a reference indicating that estimators should be be 90% confident in their estimates before using them for project planning purposes. Perhaps this can be applied in some manner.

Additional Information

The Pearson product-moment correlation coefficient article on Wikipedia describes an interpretation approach from a 1988 book (page 78) of Small, Medium, Large, and None based on the "size" of the correlation.

Any thoughts or ideas?

Answer 1

There will not be a unique answer to this question because you are trying to translate mis-rankings into some measure of value or loss and that depends on what a mis-ranking may cost the software project. Nevertheless, the scope of options is limited by some useful theoretical considerations summarized at Metrics on Permutations, a Survey (Deza & Huang).

I would suggest selecting a right-invariant metric because it does not depend on how the estimates might initially be (arbitrarily) ordered. To put it another way, we can view the correct ordering as a permutation of some fixed but arbitrary ordering of the tasks (as established by their sequence in a data file, for instance) and we need a way to measure how far an engineer's ordering departs from the correct one. We wish to do so in such a way that if the original arbitrary order were to change (e.g., the data file gets re-ordered), then (in terms of that new ordering) both the correct and estimated orderings would change, too, but our measure of distance between them ought not to change at all: that's right-invariance.

Attractive options include the $l^p$ series of metrics. Take a good look at $l^1$: this is the sum of absolute differences in ranks (perhaps appropriately scaled to lie between -1 and 1, a la Spearman). It's straightforward to compute and easy to interpret. Later, you might even want to weight this metric. For instance, it could turn out (hypothetically) that close agreement of ranks on just the top three tasks is more important than agreement with the remaining tasks. You're free to adjust your metric to reflect this.

This does not answer the question of what constitutes a "good" match between two rankings. That question is not answered with a p-value or measure of statistical significance; it is answered by understanding the impact that an incorrect estimate could have on the software project. The hope here is simply that you may find a right-invariant metric like $l^1$ to have a close association with the amount of impact. That's a matter of empirical investigation, not statistical introspection.

---

The "large, medium, small, none" assessments of correlation coefficients you may find on Wikipedia and in literature, no matter how respectable it may be, are going to be irrelevant. Ignore them. They are based on the correlations that investigators in particular fields are used to finding in the kinds of data they collect. That's meaningless to you. Even if you find such an assessment for a closely related field, such as cost estimation, it has no bearing on how you intend to use these rankings.