I'm working on a new effort estimation model (dissertation work) which involves software engineers organizing tasks (rank ordering) in order of increasing effort based solely on their understanding of the requirements. For example, writing a function to calculate an average should require less effort than writing a function to calculate an integral.

A software engineer's ability to rank order tasks in a reasonably proficient manner is essential to successfully constructing an estimate with my new model. Put another way, a software engineer should be able to get enough of the items in the correct rank order that the estimate becomes useful for project planning purposes. Some errors are expected and can be accounted for by the model (estimation is not an exact science, after all).

I believe I can use one of several rank correlation measurements to compare the software engineer's rankings to the "correct" rankings (as defined by historical data of actual effort).


How can I define reasonably proficient in terms of Spearman's ρ, Kendall's τ, etc.?

One possibility that has occurred to me is "better than random." Assume that anything better than the average of all possible arrangements indicates some level of intelligence in the ranking (ρ > 0 for Spearman, maybe?). This doesn't inspire a lot of confidence, though.

Several references talk about the statistical significance of 10%, 5%, 1%, etc., but give no guidance on why these values are selected. I can't just make up a required level of significance. I do have a reference indicating that estimators should be be 90% confident in their estimates before using them for project planning purposes. Perhaps this can be applied in some manner.

Additional Information

The Pearson product-moment correlation coefficient article on Wikipedia describes an interpretation approach from a 1988 book (page 78) of Small, Medium, Large, and None based on the "size" of the correlation.

Any thoughts or ideas?

  • $\begingroup$ Doesn't help answer your question, but software engineers are notoriously bad at estimating the effort required for a task, let alone rank ordering many of them. $\endgroup$
    – Kavka
    Feb 17, 2012 at 14:29
  • $\begingroup$ Actually, research exists that supports the ability of software engineers to produce reasonably accurate estimates in groups, hence the creation of Planning Poker, Wideband Delphi, etc. My intuition tells me that coarse rank ordering is probably easy, but fine-grained rank ordering is hard. This is why I am trying to find a measure for "reasonable proficiency." Otherwise, I'll have to do that research too! $\endgroup$ Feb 17, 2012 at 15:14
  • $\begingroup$ If you have a sufficiently group of engineers with varying levels of experience, maybe you could look at the less/medium/very experienced groups separately and then define proficiency on the basis of the overall results of the very experienced group? You could play around with some cutpoints. $\endgroup$
    – Michelle
    Feb 17, 2012 at 20:25
  • $\begingroup$ Yes, that is true. I was hoping I wouldn't be pioneering that particular bit of research, though. $\endgroup$ Feb 19, 2012 at 0:05

1 Answer 1


There will not be a unique answer to this question because you are trying to translate mis-rankings into some measure of value or loss and that depends on what a mis-ranking may cost the software project. Nevertheless, the scope of options is limited by some useful theoretical considerations summarized at Metrics on Permutations, a Survey (Deza & Huang).

I would suggest selecting a right-invariant metric because it does not depend on how the estimates might initially be (arbitrarily) ordered. To put it another way, we can view the correct ordering as a permutation of some fixed but arbitrary ordering of the tasks (as established by their sequence in a data file, for instance) and we need a way to measure how far an engineer's ordering departs from the correct one. We wish to do so in such a way that if the original arbitrary order were to change (e.g., the data file gets re-ordered), then (in terms of that new ordering) both the correct and estimated orderings would change, too, but our measure of distance between them ought not to change at all: that's right-invariance.

Attractive options include the $l^p$ series of metrics. Take a good look at $l^1$: this is the sum of absolute differences in ranks (perhaps appropriately scaled to lie between -1 and 1, a la Spearman). It's straightforward to compute and easy to interpret. Later, you might even want to weight this metric. For instance, it could turn out (hypothetically) that close agreement of ranks on just the top three tasks is more important than agreement with the remaining tasks. You're free to adjust your metric to reflect this.

This does not answer the question of what constitutes a "good" match between two rankings. That question is not answered with a p-value or measure of statistical significance; it is answered by understanding the impact that an incorrect estimate could have on the software project. The hope here is simply that you may find a right-invariant metric like $l^1$ to have a close association with the amount of impact. That's a matter of empirical investigation, not statistical introspection.

The "large, medium, small, none" assessments of correlation coefficients you may find on Wikipedia and in literature, no matter how respectable it may be, are going to be irrelevant. Ignore them. They are based on the correlations that investigators in particular fields are used to finding in the kinds of data they collect. That's meaningless to you. Even if you find such an assessment for a closely related field, such as cost estimation, it has no bearing on how you intend to use these rankings.

  • $\begingroup$ I'm not exactly looking for a measure of incorrectness, which would work into your statement about value or loss. I'm really looking for a point at which you can say that the estimator has demonstrated that they're doing more than randomly selecting an ordering. Although... I see what you're getting at, in that the "acceptable" margin of error in the estimation model could define the acceptable level of error in the rank ordering. Is that what you were alluding to? $\endgroup$ Feb 19, 2012 at 0:11
  • $\begingroup$ Yes. I would like to suggest you need something much stronger than the mere ability to determine that an engineer is doing better than random. (Even pigeons can do that. :-) How much better? How good is good enough? That's why some form of valuation has to enter into consideration. $\endgroup$
    – whuber
    Feb 19, 2012 at 15:16
  • 1
    $\begingroup$ Yes. It occurred to me over the weekend that the metric I was looking for cannot, in any way, be predicted. It must be established through experiential data. There are too many variables. For example, the data set given to the estimators will greatly affect their ability to make predictions (i.e. are the tasks separated by minutes, days, weeks, or months?). $\endgroup$ Feb 20, 2012 at 14:14
  • $\begingroup$ That's a reasonable conclusion to arrive at. Nevertheless, I believe this question should stand because (a) it is very well formulated and expressed; (b) it may serve to advance the discussion of related questions in the future; and (c) who knows?--someone may come along at any time with an excellent solution to precisely this problem, or at least with good experience to share. $\endgroup$
    – whuber
    Feb 20, 2012 at 14:27
  • $\begingroup$ Then we'll just leave it here and see what comes of it... $\endgroup$ Feb 21, 2012 at 18:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.