I'm working on a new effort estimation model (dissertation work) which involves software engineers organizing tasks (rank ordering) in order of increasing effort based solely on their understanding of the requirements. For example, writing a function to calculate an average should require less effort than writing a function to calculate an integral.
A software engineer's ability to rank order tasks in a reasonably proficient manner is essential to successfully constructing an estimate with my new model. Put another way, a software engineer should be able to get enough of the items in the correct rank order that the estimate becomes useful for project planning purposes. Some errors are expected and can be accounted for by the model (estimation is not an exact science, after all).
I believe I can use one of several rank correlation measurements to compare the software engineer's rankings to the "correct" rankings (as defined by historical data of actual effort).
How can I define reasonably proficient in terms of Spearman's ρ, Kendall's τ, etc.?
One possibility that has occurred to me is "better than random." Assume that anything better than the average of all possible arrangements indicates some level of intelligence in the ranking (ρ > 0 for Spearman, maybe?). This doesn't inspire a lot of confidence, though.
Several references talk about the statistical significance of 10%, 5%, 1%, etc., but give no guidance on why these values are selected. I can't just make up a required level of significance. I do have a reference indicating that estimators should be be 90% confident in their estimates before using them for project planning purposes. Perhaps this can be applied in some manner.
The Pearson product-moment correlation coefficient article on Wikipedia describes an interpretation approach from a 1988 book (page 78) of Small, Medium, Large, and None based on the "size" of the correlation.
Any thoughts or ideas?