11
$\begingroup$

I have a database containing a large number of experts in a field. For each of those experts i have a variety of attributes/data points like:

  • number of years of experience.
  • licenses
  • num of reviews
  • textual content of those reviews
  • The 5 star rating on each of those reviews, for a number of factors like speed, quality etc.
  • awards, assosciations, conferences etc.

I want to provide a rating to these experts say out of 10 based on their importance. Some of the data points might be missing for some of the experts. Now my question is how do i come up with such an algorithm? Can anyone point me to some relevent literature?

Also i am concerned that as with all rating/reviews the numbers might bunch up near some some values. For example most of them might end up getting an 8 or a 5. Is there a way to highlight litle differences into a larger difference in the score for only some of the attributes.

Some other discussions that i figured might be relevant:

Improve this question
$\endgroup$
1
  • $\begingroup$ It can't be done unless you come to some objective criterion; probably most of possible ratings can be constructed with some combination of your parameters. $\endgroup$
    – user88
    Commented Sep 30, 2010 at 15:39

3 Answers 3

Reset to default
12
$\begingroup$

People have invented numerous systems for rating things (like experts) on multiple criteria: visit the Wikipedia page on Multi-criteria decision analysis for a list. Not well represented there, though, is one of the most defensible methods out there: Multi attribute valuation theory. This includes a set of methods to evaluate trade-offs among sets of criteria in order to (a) determine an appropriate way to re-express values of the individual variables and (b) weight the re-expressed values to obtain a score for ranking. The principles are simple and defensible, the mathematics is unimpeachable, and there's nothing fancy about the theory. More people should know and practice these methods rather than inventing arbitrary scoring systems.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Do you know of R package for doing this? $\endgroup$
    – user333
    Commented Jun 29, 2011 at 7:35
  • 3
    $\begingroup$ @user No, and I doubt there is one. There's no magic software bullet here, by the way: almost all the work involves thinking through the issues and exploring specific trade-offs in a controlled fashion. $\endgroup$
    – whuber ♦
    Commented Jun 29, 2011 at 12:37
3
$\begingroup$

Ultimately this may not be solely a statistical exercise. PCA is a very powerful quantitative method that will allow you to generate a score or weights on its first few principal components that you can use for ranking. However, explaining what the principal components are is very challenging. They are quantitative constructs. They are not dialectic ones. Thus, to explain what they truly mean is sometimes not possible. This is especially true if you have an audience that is not quantitative. They will have no idea what you are talking about. And, will think of your PCA as some cryptic black box.

Instead, I would simply line up all the relevant variables and use a weighting system based on what one thinks the weighting should be.

I think if you develop this for outsiders, customers, users, it would be great if you could embed the flexibility of deciding on the weighting to the users.
Some users may value years of experience much more than certification and vice verse. If you can leave that decision to them. This way your algorithm is not a black box they don't understand and they are not comfortable with. You keep it totally transparent and up to them based on their own relative valuation of what matters.

Improve this answer
$\endgroup$
8
  • $\begingroup$ @Gaetan Well, for PCA you have to find a suitable numerical coding for variable such as "textual content"... $\endgroup$
    – chl
    Commented Sep 30, 2010 at 17:02
  • $\begingroup$ That's not the issue I am raising. PCA can handle dummy variables as you suggest. PCA is incredibly powerful and flexible that way. But, it is the interpretation of the principal components that gets really challenging. Let's say the first principal component starts like this: 0.02 years of experience - 0.4 textual content of reviews + 0.01 associations... Maybe you can explain it. An expert performance is proportional to years of experience, but inversely proportional to textual content of reviews? It seems absurd. But, PCA often does generate counter-intuitive results. $\endgroup$
    – Sympa
    Commented Sep 30, 2010 at 18:42
  • $\begingroup$ @Gaetan Still, I reiterate my opinion that the problem lies in how you choose to represent your variables (or how you find a useful metric). I agree with you about the difficulty of interpreting a linear combination of variables when dealing with non-continuous measurements or a mix of data types. This is why I suggested in another comment to look for alternative factorial methods. Anyway, developing scoring rules based on user preferences or expert reviewing (as is done in clinical assessment) also calls for some kind of statistical validation (at least to ensure scores reliability). $\endgroup$
    – chl
    Commented Sep 30, 2010 at 19:01
  • $\begingroup$ @Gaetan, Yes some of your comments make a lot of sense, and you're right in saying that it is not merely a statistical exercise but involves elements that are more subjective. The reason being that the intent from a user/customers standpoint might differ. Assuming he's doing a search for an expert, then i just add filters to allow him to select experts >X number of years of experience and so on But let's say he's narrowed down to 2 experts, and wants an independent comparison. So i'm just looking for a generic method to compare any two experts. $\endgroup$
    – Sidmitra
    Commented Sep 30, 2010 at 20:19
  • 2
    $\begingroup$ +1 for pointing out this is not a statistical exercise. At best, PCA can describe relationships within a particular data set and, conceivably, simplify the data by identifying near-collinearities. It is not apparent how it can inform us about how to rank the experts. $\endgroup$
    – whuber ♦
    Commented Sep 30, 2010 at 21:03
0
$\begingroup$

Do you think that you could quantify all those attributes?

If yes, I would suggest performing a principal component analysis. In the general case where all the correlations are positive (and if they aren't, you can easily get there using some transformation), the first principal component can be considered as a measure of the total importance of the expert, since it's a weighted average of all the attributes (and the weights would be the corresponding contributions of the variables - Under this perspective, the method itself will reveal the importance of each attribute). The score that each expert achieves in the first principal component is what you need to rank them.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ This looks nice, but won't it just pick highest-variance attributes and biggest clusters of cross-correlated ones? $\endgroup$
    – user88
    Commented Sep 30, 2010 at 16:21
  • 1
    $\begingroup$ Alternatively, one can perform multiple correspondence analysis or multiple factor analysis for mixed data (if numerical recoding happens to be not realistic for some variables), and the rest of your idea (computing factor scores and looking at variable loadings on the 1st dimension) applies as well. $\endgroup$
    – chl
    Commented Sep 30, 2010 at 17:16
  • 3
    $\begingroup$ It seems to me the first component will merely point out a strong direction of commonality among the experts. How could it possibly tell us who is better and who is worse, though? That requires additional information concerning the relationships between these variables and the quality of being a "good" or "bad" expert. If we believe all the variables are monotonically associated with goodness or badness, then perhaps PCA can help us explore the frontier of extreme (or maybe just outlying!) experts. Watch out though--even the monotonicity assumption is suspect. $\endgroup$
    – whuber ♦
    Commented Sep 30, 2010 at 21:09
  • 1
    $\begingroup$ @whuber I see the point, thanks. Maybe you could add this in your own response (which is very welcomed)? $\endgroup$
    – chl
    Commented Sep 30, 2010 at 21:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.