Detecting patterns of cheating and collusion in competition

Question

I'm looking for ideas on how to discover cheating in sports that have judges awarding points. I can't name the sport since I participate in it at the moment. It's been known for a long time that this sport has rampant collusion behind the scenes.

In each competition there are competitors who are evaluated on skill by a panel of judges. The spring is simple since the judges only have to place the first six competitors in rank order of preference. The thing is that sometimes the judges are actually teaching the competitor. The better the competitor is ranked in the competition, the better the judge looks and hence gets more business, and since judges know each other they have the opportunity to collude. Therein lies the incentive to cheat.

Any suggestions in how I would go about detecting if judges are fairly scoring competitors? The main constraint is that due to the size of the competitions there is not really a lot of data. Judges and competitiors would not really meet very often, four to five times at the most.

Answer 1

I can suggest a couple of paths that you might find useful. The first is my my answer to a StackOverflow question, Artificial Intelligence Methods to Detect Cheating in Games. The question was not limited to collusion but that was in fact the scope of my answer.

The second source is an excellent study by Andrew Odewahn, which i first learned in the O'Reilly compilation, Beautiful Visualization (ch 8). In this study Odewahn looks at congressional voting records for the purpose of of identifying which members of congress tend to vote together on the same bill. To do this, he creates an Affinity Matrix (which is analogous to an adjacency matrix, a common way to represent graph structures for computation.

It seemed to me from reading your Question that you either don't yet have a hypothesis regarding which judges might be colluding or you deliberately want to remain neutral on the matter and instead consider all voting patterns among all possible pairings/groupings of judges. So that's where an affinity matrix or a rendered graph might be an excellent place to begin. For instance, a rendered graph (like the one below) represents the edge weights (the number of time the two judges connected by that edge voted together) as an attractive force, hence the more two judges vote together, the closer their nodes appear in the rendered graph.

Here is one of the rendered graphs from his study (rendered in graphviz) the two different colors represent the two political parties:

Answer 2

If this is an an industry wide problem. You may want to take a look at various approaches to reader reliability testing. This is usually applied when collusion is not a fear. However, if you can identify some socially/geographically separate groups of judges, it may be possible to test whether judges that didn't know the athletes or their rankings would provide similar scores. If tape of the performance could be provided to an "independent" pool of judges and the new pool of judges was kept as blind to any additional information, then you could test if the two groups of judges returned similar scores. This would not be evidence of collusion. There could be regional differences between judges that could account for observed differences.

I know some NCAA judged sports (especially those judged by the coaches themselves) require a review of tape by a national committee before scores are excepted for entry into national level competition.