Suppose I am calculating the average rating for several products that were rated by different people.

For example, assuming that if I choose 5 random people and calculate average ratings 100 times then the variance of each average is of course higher than if I chose 10 random people. I would like to figure out how many people I would need to obtain this average with as high precision as possible.

Therefore, I believe that I can do a bootstrapping experiment where I increase the number of people whose ratings I have averaged and demonstrate that the variance decreases as I include more and more people when calculating the average(s).

This would simply amount to choosing N ratings with replacement. Taking their average. Calculating the variance across bootstrap iteration. I'm then left of a plot of variance vs. number of samples.

Does it make sense to use this plot to gain insight on how many samples I should use in order to increase the precision of my estimate? Importantly, I may only have 3 people give a rating but I want to infer that I potentially need 5 people. How would I go about inferring that?

Is there any similar work and references associated with it that I can look at?

  • $\begingroup$ With 100 x 5 observations, one can estimate mean and variance of the random variable and deduce the variance of the overall average. $\endgroup$
    – Xi'an
    Nov 3, 2022 at 8:36
  • $\begingroup$ This is a hypothetical. There aren't actually 100 x 5 observations. $\endgroup$
    – user371955
    Nov 3, 2022 at 15:11

1 Answer 1


While your general idea of estimating the variance of an estimator is correct, this probably won't give you what you actually want. If you choose N ratings with replacement your assuming that ratings are drawn from a uniform distribution which is usually not the case and if you know that it is you already know the true rating is 3 stars. If the true distribution is very concentrated around say 4 stars then even a single sample estimator would have quite low variance. If the true distribution is 50% 1 star and 50% 5 stars though, it would always have a standard deviation of exactly 2. So how do you deal with this?

The easiest option is to use the worst-case distribution I've just described as an upper bound on the variance of your estimator. If you make an assumption on the distribution over true distributions, you can actually draw distributions from that and then draw from it (instead of from a uniform distribution) and evaluate the estimator as you had proposed. For the best estimates, you can also design an estimator that first calculates the conditional distribution over true distributions given the samples it has seen so far and then evaluates the expected variance over these distributions. However, this would be a lot more involved to implement.

  • $\begingroup$ I am sampling from actual data. So for example in my actual dataset I have 3 people max for each review. I sample from these 3 with replacement for N = 1, 2 and 3. I then fit a line to the function (variance vs. N). I read this plot and infer that I need an N of ~10. Does this make sense? $\endgroup$
    – user371955
    Nov 3, 2022 at 15:15
  • $\begingroup$ Could you potentially explain how I could go about doing this assuming a 'worst-case' distribution. As mentioned assuming I have only 4 samples per review and I want to infer whether 4 is enough (has low variance) or I need more samples. Maybe when I fit a line through the function I find that 10 samples is actually what I need. $\endgroup$
    – user371955
    Nov 3, 2022 at 15:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.