I've read through a lot of answers on here but not found the answer I'm looking for the following problem - I've got a dataset of product ratings - a rating out of 5.

I've got this data for 11 different products, with the number of samples in each group as follows:

Product 1 n = 437 Product 2 n = 261 Product 3 n = 28 Product 4 n = 29 Product 5 n = 37 Product 6 n = 25 Product 7 n = 165 Product 8 n = 105 Product 9 n = 31 Product 10 n = 31 Product 11 n = 34

I want to find the best product, but can't just get all the averages because the sample sizes are so different. What process is best applied here to weight the average scores based on sample size?

I've had a look at ANOVA and run an analysis in Excel, but not sure if this is the right way to go or how to interpret the results.



Let's say you have two products with the following ratings.

Rating ID Product Value (0-5)
1 A 5
2 A 4
3 A 2
4 A 3
5 A 5
6 A 5
7 A 4
8 B 2
9 B 3
10 B 3

The overall average rating would be: $$\mu_{overall}=\frac{\sum{Value}}{n}=\frac{5+4+2+3+5+5+4+2+3+3}{10}=3.6$$

The average rating for each product would be: $$\mu_{A}=\frac{5+4+2+3+5+5+4}{7}=4$$ $$\mu_{B}=\frac{2+3+3}{3}= 2.67$$

If the product with the highest rating (Value in the example dataset) is "best" then product A is better than B. Note that the average rating for each product is weighted in the denominator (the number of times it was rated). Note that the sum of the denominators for each product will add up to the total number of ratings.

Note that if you only had the product averages, you could calculate a weighted overall average. Here the weights are equivalent to the proportion of the total ratings that the particular product got. For example,

$$\mu_{overall}=\mu_A*Wt_A+\mu_B*Wt_B=(4*\frac{7}{10})+(2.66667*\frac{3}{10}) = 3.6$$

  • $\begingroup$ Thank you! My question from here is, though, if the averages of each product are equally valid if the the sample sizes are very different? Eg: Product 1 has 437 ratings and gets an average score of 2.89 Product 10 has 31 ratings but an average score of 3.01. Can we really say Product 10 is the beter product? Does the difference in the sample size need to be considered? $\endgroup$ – eightyfish Feb 5 at 3:58
  • $\begingroup$ Yes, they are equally valid point estimates of the mean. However, the precision of the estimates will be different for each product and hence their confidence intervals different. For example, if the score are normally distributed (which may or may not be a good assumption for an ordinal variable with 6 potential values---ratings 0 through 5---something you can check by plotting their histograms), you might use the following formula for the confidence interval:statisticshowto.com/probability-and-statistics/… $\endgroup$ – Brant Inman Feb 6 at 12:38
  • $\begingroup$ Thank you! I followed your advice and found that the highest ranking product in my data (based on average score) has an average which is higher than one 95% confidence interval above the average of the product below it. I feel this means my average scores are useable? Thanks again. $\endgroup$ – eightyfish Feb 8 at 2:04
  • $\begingroup$ I am not sure what you mean by "usable". If the question is "which product has the highest average rating?", then the product with the highest average rating is the best. If the question is " Is the highest rated product statistically significantly higher than the next closest sample, you could examine the 95% confidence intervals of the two averages and, if not overlapping, they are likely statistically different. Another way to test this is to use a two sample t-test to compare both product scores. $\endgroup$ – Brant Inman Feb 9 at 3:27
  • $\begingroup$ Ok cool, that's basically what I've done. Thanks again Brant, I really appreciate your help :) $\endgroup$ – eightyfish Feb 10 at 5:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.