Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

If I have a star rating system where users can express their preference for a product or item, how can I detect statistically if the votes are highly "divided". Meaning, even if the average is 3 out of 5, for a given product, how can I detect if that is a 1-5 split versus a consensus 3, using just the data (no graphical methods)

share|improve this question
2  
What's wrong with using a Standard Deviation? –  Spork Jun 6 at 9:46
1  
Not an answer, but relevant: evanmiller.org/how-not-to-sort-by-average-rating.html –  RedSirius Jun 6 at 10:19
1  
Are you trying to detect "bimodal distribution"? See stats.stackexchange.com/q/5960/29552 –  Ben Voigt Jun 6 at 14:06
1  
In political science there is a literature on measuring political polarization that has examined various different ways of defining what is meant by "polarization." One nice paper that discusses in detail 4 different simple ways of defining polarization is the following (see pp. 692-699): educ.jmu.edu/~brysonbp/pubs/PBJ.pdf –  Jake Westfall Jun 7 at 16:02

6 Answers 6

One could construct a polarization index; exactly how one defines it depends on what constitutes being more polarized (i.e. what, exactly do you mean, in particular edge cases, by more or less polarized?):

For example, if the mean is '4', is a 50-50 split between '3' and '5' more, or less polarized than 25% '1' and 75% '5'?

Anyway, in the absence of that kind of specific definition of what you mean, I'll suggest a measure based off variance:

Given a particular mean, define the most polarized possible split as the one that maximizes variance*.

*(NB that would say that 25% '1' and 75% '5' is substantially more polarized than 50-50 split of '3's and '5's; if that doesn't match your intuition, don't use variance)

So this polarization index is the proportion of the largest possible variance (with the observed mean) in the observed variance.

Call the average rating $m$ ($m=\bar x$).

The maximum variance occurs when a proportion $p=\frac{m-1}{4}$ is at $5$ and $1-p$ is at $1$; this has a variance of $(m-1)(5-m) \cdot \frac{n}{n-1}$.

So simply take the sample variance and divide by $(m-1)(5-m) \cdot \frac{n}{n-1}$; this gives a number between $0$ (perfect agreement) and $1$ (completely polarized).

For a number of cases where the mean rating is 4, this would give the following:

enter image description here


You might instead prefer not to compute them relative to the biggest possible variance with the same mean, but instead as a percentage of the biggest possible variance for any mean rating. That would involve dividing instead by $4 \cdot \frac{n}{n-1}$, and again yields a value between 0 (perfect agreement) and $1$ (polarized at the extremes in a 50-50 ratio). This would yield the same relativities as the diagram above, but all the values would be 3/4 as large (that is, from left to right, top to bottom they'd be 0, 16.5%, 25%, 25%, 50% and 75%).

Either of the two is a perfectly valid choice - as is any other number of alternative ways of constructing such an index.

share|improve this answer
    
But then when m = 1 you get 1 - 1 = 0 and 0 / 0. How do you correct for that? –  Francesco Jun 13 at 8:17
    
@Franceso Good point. When $m=1$ or $m=5$ the formula for the first form is undefined. However, under various assumptions the limit seems to be 1, which is probably what I'd use. If someone wanted to define it as either 1 or 0 for that edge case, it would seem reasonable to me. –  Glen_b Jun 13 at 8:49

"No graphical methods" is kind of a big handicap, but...here are a couple odd ideas. Both treat the ratings as continuous, which is something of a conceptual weakness, and probably not the only one...

Kurtosis

  • The kurtosis of {1,1,1,5,5,5} = 1. You won't get a lower kurtosis with any combo of 1–5 ratings.
  • The kurtosis of {1,2,3,4,5} = 1.7. Lower means more extreme values; higher means more middle.
  • This won't work if the distribution isn't roughly symmetrical. I'll demonstrate below.

Negative binomial regression

With a data frame like this:$$\begin{array}{c|c}\rm Rating&\rm Frequency\\\hline1&31\\2&15\\3&7\\4&9\\5&37\end{array}$$Fit the model $\rm Frequency\sim\rm Rating+\sqrt{Rating}$ using negative binomial regression. The $\rm\sqrt{Rating}$ coefficient should be near zero if ratings are uniformly distributed, positive if there are proportionally more middle-range values (cf. binomial distribution), or negative with polarized distributions like the one above, for which the coefficient is -11.8.

FWIW, here's the code I've been playing around with:

x=rbinom(99,4,c(.1,.9))+1;y=sample(0:4,99,replace=T)+1 #Some polarized & uniform rating data
table(x);table(y)                                                         #Frequencies
require(moments);kurtosis(x);kurtosis(y)                                  #Kurtosis

Y=data.frame(n=as.numeric(table(y)),rating=as.numeric(levels(factor(y)))) #Data frame setup
X=data.frame(n=as.numeric(table(x)),rating=as.numeric(levels(factor(x)))) #Data frame setup
require(MASS);summary(glm.nb(n~rating+sqrt(rating),X))  #Negative binomial of polarized data
summary(glm.nb(n~rating+sqrt(rating),Y))                #Negative binomial of uniform data

Can't resist throwing in a plot...

require(ggplot2);ggplot(X,aes(x=rating,y=n))+geom_point()+stat_smooth(formula=y~x+I(sqrt(x)),method='glm',family='poisson')

The $\rm\sqrt{Rating}$ term determines the curvature (concavity in this case) of the regression line. Since I'm already cheating by using graphics, I fit this with Poisson regression instead of negative binomial because it's easier to code than doing it the right way.



Edit: Just saw this question advertised on the sidebar: and when I clicked, I saw it in the Hot Network Questions linking back to itself, as sometimes happens,

so I thought this might deserve revisiting in a more generally useful way. I decided to try my methods on the Amazon customer reviews for The Mountain Three Wolf Moon Short Sleeve Tee:

$$\begin{array}{c|ccccc}\rm Rating&1&2&3&4&5\\\hline\rm Frequency&208&54&89&198&2273\end{array}$$As you can see, this is a pretty awesome t-shirt. George Takei said so. Anyway...
The kurtosis of this distribution is quite high (7.1), so that method's not as simple as it seems.
The negative binomial regression model still works though! $\beta_\sqrt{\rm Rating}=-19.1$.

BTW, @Duncan's $\rm \sigma^2_{Frequency_\text{The Mountain Three Wolf Moon Short Sleeve Tee Ratings}}=1.31$...
and with x=rep(5:1,c(2273,198,89,54,208)), @Glen_b's polarization index var(x)/(4*length(x)/(length(x)-1))= .33 ...just sayin'.

share|improve this answer
    
It's about 0.77 for the first version of the polarization index (i.e. relative to the most polarized given the mean rating), but yes, as you say, 0.33 for the second version (relative to the most polarized distribution possible). –  Glen_b Jun 6 at 10:06
    
@Glen_b: And isn't the first version less appropriate when the mean is not fixed across various sets of ratings that need comparing? Or have I misunderstood your answer? –  Nick Stauner Jun 6 at 17:29
    
It depends on what the aim is. Judging by the title "how to detect polarized opinions", I lean toward the first (given the average rating, how polarized is opinion about that?). If the aim was indeed to compare different sets of ratings, it may make more sense to work with the second approach, as you suggest. Its why I did both. My comment was not intended in any sense as criticism; I'm flattered you mentioned it at all. –  Glen_b Jun 7 at 2:30
    
@Glen_b: Understood :) TBH, I am suggesting the negative binomial regression modeling approach is better, but I admit I've hardly tested it thoroughly. I have a feeling most real polarized rating sets won't be evenly polarized, so I'm thinking robustness against asymmetry is going to be important for future readers. –  Nick Stauner Jun 7 at 3:01

I would think an easy way is to calculate the variance. In a simple system like that, a higher variance would mean more 1s/5s. EDIT Quick example: if your values are 1,3,3,5 your variance will be: $$\frac {(1-3)^2 + (3-3)^2 + (3-3)^2 + (5-3)^2}4 = 1$$If your numbers are 1,1,5,5 your variance will be:$$\frac {(1-3)^2 + (1-3)^2 + (5-3)^2 + (5-3)^2}4 = 2$$

share|improve this answer

I doubt that I can add something valuable to the clever answers already given. In particular, to @Glen_b's fine idea to assess how the variance observed is relatively close to the maximal variance possible under the observed mean. My own blunt and straight from the shoulder proposal is, instead, about some robust measure of dispersion based not on deviations from some centre but directly on distances between data points.

Compute pairwise distances (absolute differences) between all the data points. Drop out $d_{ii}$ zero distances. Compute a central tendency in the distribution of the distances (the choice is yours; it may be, for example, mean, median, or Hodges-Lehmann centre).

Rating scale                   Distances      Mean     Median    Hodges-Lehmann
1  2  3  4  5

Frequency distributions:

1     2     1                 0 2 2 2 2 4      2          2          2

2           2                 0 0 4 4 4 4      2.7        4          2

1        2  1                 0 1 1 3 3 4      2          2          2

1  1  1     1                 1 1 2 2 3 4      2.2        2          2

1  1     1  1                 1 1 2 3 3 4      2.3        2.5        2.5

1           3                 0 0 0 4 4 4      2          2          2

As you can see, the 3 statistics may be very different as measures of "polarization" (if I were to measure "disagreement" rather than bipolar confrontation, I would probably choose HL). The choice is yours. One notion: if you compute squared distances, their mean will be directly related to usual variance in the data (and so you will arrive at @Duncan's suggestion to compute variance). Computation of distances won't be too hard even with big $N$ here because the rating scale is descrete and with relatively few grades, so frequency-weighting algorithm to compute distances offers itself naturally.

share|improve this answer
    
The mean of the pairwise squared distances is related to the variance. –  Glen_b Jun 6 at 10:09

How about, if the 3 star rating is smaller than the average of the 5 and 4, and also smaller than the average of the 1 and 2:

if (number_of_ratings > 6)      // kind of meaningless unless there's enough ratings
{
    if ( ((rating(5)+rating(4))*0.5 > rating(3)) &&
         ((rating(1)+rating(2))*0.5 > rating(3))
       )    
    {
        // Opinion divided
    }
    else
    {
        // Opinion not divided
    }
}
else
{
    // Hard to tell yet if opinion is divided
}

Off the top of my head I can't think of any situation in which that wouldn't work. Using the example above: Amazon customer reviews for The Mountain Three Wolf Moon Short Sleeve Tee:

$$\begin{array}{c|ccccc}\rm Rating&1&2&3&4&5\\\hline\rm Frequency&208&54&89&198&2273\end{array}$$

In this case:

$$\begin{array}{c|ccccc}\rm Rating&average(1,2)&3&average(4,5)\\\hline\rm Frequency&131&89&1235\end{array}$$

This would pass the test and be considered divided opinion.

share|improve this answer
1  
what if there were lots of 2s and 4s, and relatively few other ratings? It is hard to imagine this happening in reality, but would one really want to call that polarized? –  Nick Stauner Jun 7 at 3:09
    
Come to think of it, one could more easily find cases with lots of 1s and 5s, very few 2s and 4s, and a moderate amount of 3s. For instance, $$\begin{array}{c|ccccc}\rm Rating&1&2&3&4&5\\\hline\rm Frequency&25&5&15&5&25\end{array}$$ That's pretty polarized, no? Yet your method would produce the same result for this as for a uniform distribution of 15 of each rating. –  Nick Stauner Jun 7 at 4:03

I think what you are looking for is standard deviation: $$ \sigma = \sqrt{\frac{\sum_{i=0}^{n}(x_i-\mu )^2}{n}}\\\text{where }\sigma \text{ is standard deviation, } \\ n \text{ is the number of data points,}\\ x \text{ represents all of the data points, and}\\\mu\text{ is the mean.} $$

I don't know what programming language this is, but here's a java method that will give you standard deviation:

public static double standardDeviation(double[] data) {
            //find the mean
    double sum = 0;
    for(double x:data) {
        sum+=x;
    }
    double mean = sum/data.length;

            //find standard deviation
    Double sd;
    sd=0.0;
    for(double x:data) {
        sd+=Math.pow((x-mean),2);
    }
    sd=sd/data.length;
    sd=Math.sqrt(sd);

    return sd;
}
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.