Figure out the number of samples needed to increase the precision of an estimate

Question

Suppose I am calculating the average rating for several products that were rated by different people.

For example, assuming that if I choose 5 random people and calculate average ratings 100 times then the variance of each average is of course higher than if I chose 10 random people. I would like to figure out how many people I would need to obtain this average with as high precision as possible.

Therefore, I believe that I can do a bootstrapping experiment where I increase the number of people whose ratings I have averaged and demonstrate that the variance decreases as I include more and more people when calculating the average(s).

This would simply amount to choosing N ratings with replacement. Taking their average. Calculating the variance across bootstrap iteration. I'm then left of a plot of variance vs. number of samples.

Does it make sense to use this plot to gain insight on how many samples I should use in order to increase the precision of my estimate? Importantly, I may only have 3 people give a rating but I want to infer that I potentially need 5 people. How would I go about inferring that?

Is there any similar work and references associated with it that I can look at?

Answer 1

While your general idea of estimating the variance of an estimator is correct, this probably won't give you what you actually want. If you choose N ratings with replacement your assuming that ratings are drawn from a uniform distribution which is usually not the case and if you know that it is you already know the true rating is 3 stars. If the true distribution is very concentrated around say 4 stars then even a single sample estimator would have quite low variance. If the true distribution is 50% 1 star and 50% 5 stars though, it would always have a standard deviation of exactly 2. So how do you deal with this?

The easiest option is to use the worst-case distribution I've just described as an upper bound on the variance of your estimator. If you make an assumption on the distribution over true distributions, you can actually draw distributions from that and then draw from it (instead of from a uniform distribution) and evaluate the estimator as you had proposed. For the best estimates, you can also design an estimator that first calculates the conditional distribution over true distributions given the samples it has seen so far and then evaluates the expected variance over these distributions. However, this would be a lot more involved to implement.