I recently started analyzing episode polling data from a website that uses a 1-10 rating system. I've been reading about ranking star rating systems using Bayesian Credible Interviews as explained by Evan Miller. Using a Python implementation of his logic, I've been able to derive the lower (and upper) credible interval bounds for each episode's rating at the 90% confidence level.
In the article linked above, Evan says sorting by the lower bound is an abridged approach to ranking the items. When I do this, however, I feel that some of the items may be sorted improperly because the credible intervals overlap. For example, should more narrow intervals take precedence if the lower bound scores are similar?
Additionally, he states:
A more rigorous approach to the sorting problem would minimize a loss function, perhaps a multilinear loss function, over the Dirichlet distribution.
^ This is pretty much a foreign language to me! How would/could one implement a more robust ranking system (using Python specifically)? I created a Google Sheet with the score distributions for each episode poll and calculations for the mean/intervals. I also included a graph illustrating the overlapping intervals issue.
Any help is much appreciated. Thanks!