I'm trying to come up with a measure of how "divisive" a set of values that correspond to user-ratings on a 1-5 scale are. What I mean is I want to reward distributions that have BOTH high frequencies of ones and high frequencies of fives and low frequencies of twos, threes and fours. A distribution with only a high frequency of ones but few fives should get a lower score, as the 'ideal' distribution I'm comparing against is strongly bimodal.
For an n-sized sample I currently create a fake distribution that has n/2 ones and n/2 fives (as well as ensure that each category has a minimum frequency of 5, as suggested by the Python documentation for chi-square tests). I then get the chi-squared statistic between this fake, "perfect" distribution and the sampled data and use this value to obtain a score.
I'm using Python, specifically scipy.stats.chisqprob, and the actual score measure I implemented is the absolute value of the log of the result of that function (using 4 degrees of freedom).
I'm getting reasonable results, so something like this has a score of 20 (the lower the score, the closer the distribution is to the "perfect" one):
Whereas a mostly-uniform distribution has a very high score of 160, as I'd expect.
Now, my problem is when the distribution has lots of ones, but few fives (or vice versa), as this still gets reasonably good scores. Take this example, the score being around 45, which is still good when compared with all the other results in general:
Is the chi-squared test appropriate for what I am trying to do? If so, what sort of things could I tweak to try and improve upon my results?