1
$\begingroup$

I have 2 movies, their rating distribution is this:

movie_id 1_star 2_star  3_star  4_star  5_star  std
1        0.661  0.231   0.081   0.012   0.015   1.629
2        0.002  0.005   0.071   0.314   0.608   1.335

1~5 star corresponding to the rating as 2~10

If we look at the distribution, these 2 would look like

enter image description here

enter image description here

The question is that, which one would have a higher deviation in the rating?

While the first movie's rating looks more clustered, but it turns out that it has a higher standard deviation.

What is happening?


My analysis is that, notice the tail distribution, the movie 1 has 1.4% with 5 star, while movie 2 only has 0.2%.

But I'm not sure if this is the reason? or there are other reasons?

$\endgroup$

1 Answer 1

2
$\begingroup$

If you calculate the contribution to the sum of squares in the numerator of the variance calculation, you can see that the difference in the category most distant from the mode accounts for a substantial fraction of the difference in variance (roughly 2/3 of it), so your guess it largely correct.

The next main contribution comes from the next category in.

The means are similar enough that the effect of distance from the mean is small, it's mainly differences in proportion impacting the difference in contribution to the variance (and so the standard deviation).

$\endgroup$
4
  • $\begingroup$ Yes, that's the same calculation that I've done. but it's a bit counter-intuitive that such a tiny portion of high rating would affect the std dramatically. Is there any good/detailed explanation? $\endgroup$
    – ZK Zhao
    May 23, 2017 at 10:34
  • $\begingroup$ The distance from the mean is squared in the calculation of variance. This weights that difference in proportion in the most distant category much more heavily than a similar difference in the modal category -- about 50 times as much. $\endgroup$
    – Glen_b
    May 23, 2017 at 10:42
  • $\begingroup$ And is there any alternative than the std to describe the extent of how data clustering together? I think std is standard and there is no other good way to do it? $\endgroup$
    – ZK Zhao
    May 23, 2017 at 10:46
  • 1
    $\begingroup$ It depends on what you're trying to pick up; there are many ways to measure something like standard deviation, so you'd need to clarify what kind of thing you're trying to measure that's not standard deviation. That might be a new question though. $\endgroup$
    – Glen_b
    May 23, 2017 at 10:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.