Timeline for How do I order or rank a set of experts?
Current License: CC BY-SA 2.5
11 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Sep 30, 2010 at 23:34 | vote | accept | Sidmitra | ||
| Oct 1, 2010 at 21:33 | |||||
| Sep 30, 2010 at 21:03 | comment | added | whuber♦ | +1 for pointing out this is not a statistical exercise. At best, PCA can describe relationships within a particular data set and, conceivably, simplify the data by identifying near-collinearities. It is not apparent how it can inform us about how to rank the experts. | |
| Sep 30, 2010 at 20:43 | comment | added | Sympa | @Sidmitra, your comments make sense. Maybe you will derive good use out of PCA after all. I would suggest you may run PCA several times by deriving the simplest and most explainable models that still explains most of the variance between experts. By doing so, hopefully you may end up with a PCA model with principal components that are readily explainable to others and to yourself. | |
| Sep 30, 2010 at 20:30 | comment | added | Sidmitra | @chl numerical coding for "textual content", could be as simple as some sort of sentiment analysis and giving them a binary value of good/bad? | |
| Sep 30, 2010 at 20:26 | comment | added | Sidmitra | @Gaetan, So yes i'm assuming i can somehow quantify all of the attributes, which again might be subjective. But anyhow i see some interesting points in this discussion and will think some more on that before commenting further. | |
| Sep 30, 2010 at 20:19 | comment | added | Sidmitra | @Gaetan, Yes some of your comments make a lot of sense, and you're right in saying that it is not merely a statistical exercise but involves elements that are more subjective. The reason being that the intent from a user/customers standpoint might differ. Assuming he's doing a search for an expert, then i just add filters to allow him to select experts >X number of years of experience and so on But let's say he's narrowed down to 2 experts, and wants an independent comparison. So i'm just looking for a generic method to compare any two experts. | |
| Sep 30, 2010 at 19:01 | comment | added | chl | @Gaetan Still, I reiterate my opinion that the problem lies in how you choose to represent your variables (or how you find a useful metric). I agree with you about the difficulty of interpreting a linear combination of variables when dealing with non-continuous measurements or a mix of data types. This is why I suggested in another comment to look for alternative factorial methods. Anyway, developing scoring rules based on user preferences or expert reviewing (as is done in clinical assessment) also calls for some kind of statistical validation (at least to ensure scores reliability). | |
| Sep 30, 2010 at 18:42 | comment | added | Sympa | That's not the issue I am raising. PCA can handle dummy variables as you suggest. PCA is incredibly powerful and flexible that way. But, it is the interpretation of the principal components that gets really challenging. Let's say the first principal component starts like this: 0.02 years of experience - 0.4 textual content of reviews + 0.01 associations... Maybe you can explain it. An expert performance is proportional to years of experience, but inversely proportional to textual content of reviews? It seems absurd. But, PCA often does generate counter-intuitive results. | |
| Sep 30, 2010 at 18:39 | history | edited | Sympa | CC BY-SA 2.5 |
added 274 characters in body
|
| Sep 30, 2010 at 17:02 | comment | added | chl | @Gaetan Well, for PCA you have to find a suitable numerical coding for variable such as "textual content"... | |
| Sep 30, 2010 at 16:15 | history | answered | Sympa | CC BY-SA 2.5 |