I'm trying to come up with a measure of how "divisive" a set of values that correspond to user-ratings on a 1-5 scale are. What I mean is I want to reward distributions that have BOTH high frequencies of ones and high frequencies of fives and low frequencies of twos, threes and fours. A distribution with only a high frequency of ones but few fives should get a lower score, as the 'ideal' distribution I'm comparing against is strongly bimodal.

For an n-sized sample I currently create a fake distribution that has n/2 ones and n/2 fives (as well as ensure that each category has a minimum frequency of 5, as suggested by the Python documentation for chi-square tests). I then get the chi-squared statistic between this fake, "perfect" distribution and the sampled data and use this value to obtain a score.

I'm using Python, specifically scipy.stats.chisqprob, and the actual score measure I implemented is the absolute value of the log of the result of that function (using 4 degrees of freedom).

I'm getting reasonable results, so something like this has a score of 20 (the lower the score, the closer the distribution is to the "perfect" one):

A good divisiveness score

Whereas a mostly-uniform distribution has a very high score of 160, as I'd expect.

Now, my problem is when the distribution has lots of ones, but few fives (or vice versa), as this still gets reasonably good scores. Take this example, the score being around 45, which is still good when compared with all the other results in general:

A divisiveness score that should be much worse than it is

Is the chi-squared test appropriate for what I am trying to do? If so, what sort of things could I tweak to try and improve upon my results?

  • $\begingroup$ The beta-binomial distribution has been used, e.g., in publishing to model the "bathtub" or u-shaped distribution of magazine readership where you will have a lot of readers who read only 1 issue of a monthly publication and a big bunch of other readers (subscribers) who read it on a regular, monthly basis. Here's a link to its description on Wikipedia ... en.wikipedia.org/wiki/Beta-binomial_distribution In the graph of the Probability mass function the example where *a*=0.2 and *b*=0.25 comes closest to what you're describing. $\endgroup$ – Mike Hunter Nov 17 '15 at 17:38
  • $\begingroup$ Will you always have the same amount of data in every distribution to compare to the ideal? Thinking of this generically as a kind of correlation between a realized distribution and an ideal one, note that there are lots of different intermediate patterns that will be ranked differently using different measures of correlation / similarity. $\endgroup$ – gung - Reinstate Monica Nov 17 '15 at 17:38
  • $\begingroup$ @gung, you're right, there are too many intermediate patterns that give unexpected rankings. I think the suggestion about standard deviation is a simpler way to go. $\endgroup$ – dasboth Nov 18 '15 at 21:47

The chi-squared test only tests whether an association exists between your categorical variables; it doesn't test for a particular association. There may be a derivative of the chi-squared test designed for such a measure, but I can't come up with one off the top of my head.

I might argue that standard deviation/variance might be appropriate here. Though these are designed for continuous data rather than Likert data, the SD/variance of a data set will be minimized if all data are identical and will be maximized if there are two equally-sized peaks at 1 and 5. I think this is likely what you're looking for. You can also calculate the standard deviation of two sets of data separately and conduct a hypothesis test to see if one is significantly different from the other.

  • $\begingroup$ I agree. Standard deviation is a good start here. $\endgroup$ – Nick Cox Nov 17 '15 at 17:39
  • 1
    $\begingroup$ Aren't we looking at a goodness of fit chi-square, rather than a test of association here? (Or did I miss something ?) $\endgroup$ – Glen_b -Reinstate Monica Nov 18 '15 at 1:59
  • $\begingroup$ @Matt, I think you're right about the standard deviation. I was obviously overthinking it a bit. I've implemented a new version that relies on the s.d. I'll need to scale it somehow so it's relative to the ideal s.d. of around 2 but apart from that, my initial tests look promising! $\endgroup$ – dasboth Nov 18 '15 at 21:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.