How do I order or rank a set of experts? I have a database containing a large number of experts in a field. For each of those experts i have a variety of attributes/data points like:


*

*number of years of experience.

*licenses

*num of reviews

*textual content of those reviews

*The 5 star rating on each of those reviews, for a number of factors like speed, quality etc.

*awards, assosciations, conferences etc.


I want to provide a rating to these experts say out of 10 based on their importance. Some of the data points might be missing for some of the experts. Now my question is how do i come up with such an algorithm? Can anyone point me to some relevent literature? 
Also i am concerned that as with all rating/reviews the numbers might bunch up near some some values. For example most of them might end up getting an 8 or a 5. Is there a way to highlight litle differences into a larger difference in the score for only some of the attributes.
Some other discussions that i figured might be relevant:


*

*Bayesian rating system with multiple categories for each rating

*How would YOU compute IMDB movie rating?

*Eliciting priors from experts

*What are some of the best ranking algorithms with inputs as up and down votes?
 A: Ultimately this may not be solely a statistical exercise.  PCA is a very powerful quantitative method that will allow you to generate a score or weights on its first few principal components that you can use for ranking.  However, explaining what the principal components are is very challenging.  They are quantitative constructs.  They are not dialectic ones.  Thus, to explain what they truly mean is sometimes not possible.  This is especially true if you have an audience that is not quantitative.  They will have no idea what you are talking about.  And, will think of your PCA as some cryptic black box.    
Instead, I would simply line up all the relevant variables and use a weighting system based on what one thinks the weighting should be.
I think if you develop this for outsiders, customers, users, it would be great if you could embed the flexibility of deciding on the weighting to the users.
Some users may value years of experience much more than certification and vice verse.  If you can leave that decision to them.  This way your algorithm is not a black box they don't understand and they are not comfortable with.  You keep it totally transparent and up to them based on their own relative valuation of what matters.
A: People have invented numerous systems for rating things (like experts) on multiple criteria: visit the Wikipedia page on Multi-criteria decision analysis for a list.  Not well represented there, though, is one of the most defensible methods out there: Multi attribute valuation theory.    This includes a set of methods to evaluate trade-offs among sets of criteria in order to (a) determine an appropriate way to re-express values of the individual variables and (b) weight the re-expressed values to obtain a score for ranking.  The principles are simple and defensible, the mathematics is unimpeachable, and there's nothing fancy about the theory.  More people should know and practice these methods rather than inventing arbitrary scoring systems.
A: Do you think that you could quantify all those attributes? 
If yes, I would suggest performing a principal component analysis. In the general case where all the correlations are positive (and if they aren't, you can easily get there using some transformation), the first principal component can be considered as a measure of the total importance of the expert, since it's a weighted average of all the attributes (and the weights would be the corresponding contributions of the variables - Under this perspective, the method itself will reveal the importance of each attribute). The score that each expert achieves in the first principal component is what you need to rank them. 
