If I have a population of exactly 100.000 voters, and I know they all voted in either candidate A, B, C, D or E. (e.g where the results of a random sample of 100 voters are A = 40%, B = 30%, C = 10%, D = 10%, E = 10%).

If I incrementally increase my sample size to 200 people, count the votes, then to 300, 400, etc, I get a more accurate prediction with increasing sample size. However, once larger samples are more costly; what test/metric can I use on random samples to determine what is the smallest sample size that allows me to say:

"with this population size (voters) and this number of options (candidates), this sample size of x has a 95% probability of detecting the winner (e.g A>B)"

Common sense tells me that if the final election result is A=50% B=49%, the sample needed to find the winner will be much bigger than if it is A=90%, B=9%. But I think there must be a test that looks into how incremental sample sizes impact the results, and might be used to inform me that after a certain point increasing the sample size is unlikely to change the reliability of my prediction.

  • 1
    $\begingroup$ Try doing a search (on Google or this site) for power analysis. $\endgroup$
    – tddevlin
    Aug 16, 2018 at 17:28
  • $\begingroup$ Are you sampling with or without replacement? $\endgroup$
    – Sebastian
    Aug 16, 2018 at 20:20
  • 1
    $\begingroup$ @tddevlin Although power analysis is relevant, it's not going to give a useful answer in this sequential testing situation. The key difference (IMHO) is this question is explicitly aware that the effect size and variances which must be input into any power calculation are themselves uncertain estimates based on the initial sample, and so that uncertainty needs to be accommodated. $\endgroup$
    – whuber
    Aug 16, 2018 at 21:22
  • $\begingroup$ Orangetree: The samples would be without replacement, because once someone voting choice is known, there is no point in making it unknown again. Information in this case would be cumulative (e.g. Get data on 100 people, count votes, get 100 more, count votes, get 100 more, etc.). $\endgroup$ Aug 17, 2018 at 17:09
  • $\begingroup$ Whuber: I didn't explicitly mention variance, but I believe that is a key factor in solving my problem. How can I accommodate for unknown effect size and for the inherent uncertainty of the variance in the initial sample? $\endgroup$ Aug 17, 2018 at 17:20


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.