# How do I order or rank a set of experts?

I have a database containing a large number of experts in a field. For each of those experts i have a variety of attributes/data points like:

• number of years of experience.
• num of reviews
• textual content of those reviews
• The 5 star rating on each of those reviews, for a number of factors like speed, quality etc.
• awards, assosciations, conferences etc.

I want to provide a rating to these experts say out of 10 based on their importance. Some of the data points might be missing for some of the experts. Now my question is how do i come up with such an algorithm? Can anyone point me to some relevent literature?

Also i am concerned that as with all rating/reviews the numbers might bunch up near some some values. For example most of them might end up getting an 8 or a 5. Is there a way to highlight litle differences into a larger difference in the score for only some of the attributes.

Some other discussions that i figured might be relevant:

-
It can't be done unless you come to some objective criterion; probably most of possible ratings can be constructed with some combination of your parameters. –  mbq Sep 30 '10 at 15:39

People have invented numerous systems for rating things (like experts) on multiple criteria: visit the Wikipedia page on Multi-criteria decision analysis for a list. Not well represented there, though, is one of the most defensible methods out there: Multi attribute valuation theory. This includes a set of methods to evaluate trade-offs among sets of criteria in order to (a) determine an appropriate way to re-express values of the individual variables and (b) weight the re-expressed values to obtain a score for ranking. The principles are simple and defensible, the mathematics is unimpeachable, and there's nothing fancy about the theory. More people should know and practice these methods rather than inventing arbitrary scoring systems.

-
(+1) and thanks for the references. –  chl Oct 1 '10 at 6:50
Do you know of R package for doing this? –  user333 Jun 29 '11 at 7:35
@user No, and I doubt there is one. There's no magic software bullet here, by the way: almost all the work involves thinking through the issues and exploring specific trade-offs in a controlled fashion. –  whuber Jun 29 '11 at 12:37

Ultimately this may not be solely a statistical exercise. PCA is a very powerful quantitative method that will allow you to generate a score or weights on its first few principal components that you can use for ranking. However, explaining what the principal components are is very challenging. They are quantitative constructs. They are not dialectic ones. Thus, to explain what they truly mean is sometimes not possible. This is especially true if you have an audience that is not quantitative. They will have no idea what you are talking about. And, will think of your PCA as some cryptic black box.

Instead, I would simply line up all the relevant variables and use a weighting system based on what one thinks the weighting should be.

I think if you develop this for outsiders, customers, users, it would be great if you could embed the flexibility of deciding on the weighting to the users.
Some users may value years of experience much more than certification and vice verse. If you can leave that decision to them. This way your algorithm is not a black box they don't understand and they are not comfortable with. You keep it totally transparent and up to them based on their own relative valuation of what matters.

-
@Gaetan Well, for PCA you have to find a suitable numerical coding for variable such as "textual content"... –  chl Sep 30 '10 at 17:02
That's not the issue I am raising. PCA can handle dummy variables as you suggest. PCA is incredibly powerful and flexible that way. But, it is the interpretation of the principal components that gets really challenging. Let's say the first principal component starts like this: 0.02 years of experience - 0.4 textual content of reviews + 0.01 associations... Maybe you can explain it. An expert performance is proportional to years of experience, but inversely proportional to textual content of reviews? It seems absurd. But, PCA often does generate counter-intuitive results. –  Gaetan Lion Sep 30 '10 at 18:42
@Gaetan Still, I reiterate my opinion that the problem lies in how you choose to represent your variables (or how you find a useful metric). I agree with you about the difficulty of interpreting a linear combination of variables when dealing with non-continuous measurements or a mix of data types. This is why I suggested in another comment to look for alternative factorial methods. Anyway, developing scoring rules based on user preferences or expert reviewing (as is done in clinical assessment) also calls for some kind of statistical validation (at least to ensure scores reliability). –  chl Sep 30 '10 at 19:01
@Gaetan, Yes some of your comments make a lot of sense, and you're right in saying that it is not merely a statistical exercise but involves elements that are more subjective. The reason being that the intent from a user/customers standpoint might differ. Assuming he's doing a search for an expert, then i just add filters to allow him to select experts >X number of years of experience and so on But let's say he's narrowed down to 2 experts, and wants an independent comparison. So i'm just looking for a generic method to compare any two experts. –  Sidmitra Sep 30 '10 at 20:19
+1 for pointing out this is not a statistical exercise. At best, PCA can describe relationships within a particular data set and, conceivably, simplify the data by identifying near-collinearities. It is not apparent how it can inform us about how to rank the experts. –  whuber Sep 30 '10 at 21:03