# How could I improve upon a chi-square test to measure the similarity of two distributions?

I'm trying to come up with a measure of how "divisive" a set of values that correspond to user-ratings on a 1-5 scale are. What I mean is I want to reward distributions that have BOTH high frequencies of ones and high frequencies of fives and low frequencies of twos, threes and fours. A distribution with only a high frequency of ones but few fives should get a lower score, as the 'ideal' distribution I'm comparing against is strongly bimodal.

For an n-sized sample I currently create a fake distribution that has n/2 ones and n/2 fives (as well as ensure that each category has a minimum frequency of 5, as suggested by the Python documentation for chi-square tests). I then get the chi-squared statistic between this fake, "perfect" distribution and the sampled data and use this value to obtain a score.

I'm using Python, specifically scipy.stats.chisqprob, and the actual score measure I implemented is the absolute value of the log of the result of that function (using 4 degrees of freedom).

I'm getting reasonable results, so something like this has a score of 20 (the lower the score, the closer the distribution is to the "perfect" one): Whereas a mostly-uniform distribution has a very high score of 160, as I'd expect.

Now, my problem is when the distribution has lots of ones, but few fives (or vice versa), as this still gets reasonably good scores. Take this example, the score being around 45, which is still good when compared with all the other results in general: Is the chi-squared test appropriate for what I am trying to do? If so, what sort of things could I tweak to try and improve upon my results?

• The beta-binomial distribution has been used, e.g., in publishing to model the "bathtub" or u-shaped distribution of magazine readership where you will have a lot of readers who read only 1 issue of a monthly publication and a big bunch of other readers (subscribers) who read it on a regular, monthly basis. Here's a link to its description on Wikipedia ... en.wikipedia.org/wiki/Beta-binomial_distribution In the graph of the Probability mass function the example where *a*=0.2 and *b*=0.25 comes closest to what you're describing. – Mike Hunter Nov 17 '15 at 17:38
• Will you always have the same amount of data in every distribution to compare to the ideal? Thinking of this generically as a kind of correlation between a realized distribution and an ideal one, note that there are lots of different intermediate patterns that will be ranked differently using different measures of correlation / similarity. – gung - Reinstate Monica Nov 17 '15 at 17:38
• @gung, you're right, there are too many intermediate patterns that give unexpected rankings. I think the suggestion about standard deviation is a simpler way to go. – dasboth Nov 18 '15 at 21:47