Consider a system like Tinder for example where users can vote yes or no to a profile (like or dislike). If Tinder wanted to come up with an internal percentile rating of a user based only on the other users' votes of their profile (not based on how picky that particular user is), what would be a good statistical approach?
I have thought a good amount about this and I think there are two important points to consider.
1) Different users have different styles of voting. Some users are pickier than others. Thus, a like from a picky user should impact a profile's percentile more positively than a like from a less picky user. The same logic goes for dislikes. 2) Votes more likely to be spam should be weighted less heavily in the percentile. Spam can be determined by somehow analyzing a user's voting style and compare each vote they do to what everyone else is voting for a specific profile.
Beyond this I'm struggling to come up with a good algorithm to do this.
One other issue I'm thinking about is that let's say you come up with some number that represents a user's score in the system. You can get this user's percentile by getting the ranking of that user's score among all the scores. However, what if your system only has a small number of users? The system I am thinking about is supposed to predict globally what the profile's percentile would be. Thus, if there are only 5 profiles on the system and it is the top profile, this will only be the top 20th percentile. This would only be a problem when the system only has a few users, but I am wondering how to represent some idea of a percentile in this case.