How to adjust for Benford's law and recency in playlist?

Question

I have a private SoundCloud playlist of songs that I have made available to a group of people. I am trying to work out which of the songs is the "best", in some sense.

By definition, the play count of each song in the playlist is the popularity of the song.

However I want to model the tracks' individual play counts as being in some way related to the "goodness" of the tracks.

As I understand it, because of Benford's Law / the fact that if someone just clicks "play" on the entire playlist, and then stops at any time, the earlier tracks in the playlist receive disproportionately many plays.

How can I adjust for this effect so that I can more accurately compare the play counts of the different songs? Could I run an experiment with identical tracks and then from that derive a distribution that I can subtract to get the "goodness" values?

Also there's another confounding factor which is that I add another song every week (I write the songs), which means that the older songs get more plays. But I guess I can just divide by "days since upload date" at some point to remove that from my "goodness" assessment.

Answer 1

The best option is to randomize the order of the tracks that you send to each person. This would be the gold standard for experimental design, and should cancel out any Benford's law effects if you have a large enough sample size.

If it's not feasible to give everyone their own track list, you might make 10 (randomly sorted) track lists, distribute those to your test subjects, and use a mixed-effects model, where you have some fixed effects related to the order of the track in the playlist, and then random effects from the "underlying quality" of each track.

http://www.r-bloggers.com/linear-mixed-models-in-r/

https://github.com/michaelbilow/pylmm_zarlab [Disclaimer: this is a project I work on, and it's primarily for biological data, but if you're inclined, the lmm.py script can be excerpted to do mixed models in Python if you're more comfortable.]