A problem I've been thinking of and want to make sure I'm on the right track. It's somewhat similar to this question I think: Binomial Probability Question. But I was curious if I could approach it from a different direction.

Say I'm picking between two basketball players for my team. I want to evaluate them based only on their make percentage from $n$ free throws taken during the tryout. How many shots should I have them take to have some level of certainty in my selection of who is the better shooter?

My initial thought is that I could answer this via power analysis. If I assume a probability of Type I error, $\alpha=0.05$, probability of Type II error, $\beta=0.2$, and assume that make% of player 1 $P_1 = 0.5$ and make% of player 2 $P_2= 0.6$, then what does my sample size $n$ need to be?

I plug this information into G*Power (Proportions: Inequality, two independent groups (Fisher's exact test), two-tailed) and get a sample size of $n=404$ shots.

Is this a valid way to approach this problem? Can you think of a better or more generalizable way?

  • 1
    $\begingroup$ Interesting question. My idea would also to be do a variety of power analyses and see at what level of certainty you are OK with. But this also feels similar to the multi-armed bandit problem to me, which you might find interesting and relevant here: en.wikipedia.org/wiki/Multi-armed_bandit $\endgroup$ – Mark White Apr 1 '20 at 22:10
  • $\begingroup$ @MarkWhite Thanks for the direction. I hadn't heard of that problem before. Looks interesting! $\endgroup$ – user279473 Apr 1 '20 at 22:15
  • $\begingroup$ I think in this problem is difficult to make a precise prediction because the number of shots necessary depends not only on the difference between the two but also on the overall accuracy level (disentangling 0.5 from 0.55 is not the same as disentangling 0.95 from 1). You could rephrase the problem as asking: what minimum number of shots is necessary to achieve on average a certain level of confidence? This can be answered but requires choosing some prior probability distribution for the accuracy of players. $\endgroup$ – matteo Apr 2 '20 at 12:08
  • $\begingroup$ @matteo Yeah, I realized that the initial accuracy level definitely has an effect on the answer. That's actually why I selected 0.5 and 0.6 as that is kind of the worst case scenario considering a delta of 0.1. Looking for an average level of confidence is a good middle ground I think. Thanks for your thoughts. $\endgroup$ – user279473 Apr 2 '20 at 13:23
  • $\begingroup$ @user279473 my point is that you don't need to commit to a single initial accuracy level, but you can "average" the required number of trials over the prior distribution about the players' accuracy level. $\endgroup$ – matteo Apr 2 '20 at 14:57

Comments: Your answer seems about right. Here are results taking a couple of different points of view, and giving answers similar to yours.

With $p=0.5,$ the margin of error of a 95% CI for $p$ based on $n = 400$ observations would be $1.96\sqrt{.5^2/400} \approx .05.$

Similarly, with $p=.6$ the margin of error with $n=400$ is about the same.

So with $n = 400,$ you should rarely have trouble distinguishing $p_1 = 0.5 \pm 0.05$ from $p_2 = 0.6 \pm 0.05$ fractions of successful free throws.

Computations in R:

[1] 0.049
[1] 0.04801

Minitab's Power and Sample Size procedure for a test of differences in two proportions gives the following result for distinguishing between $.5$ and $.6$ at the 5% level with power .8. (The test uses normal approximations.)

Power and Sample Size 

Test for Two Proportions

Testing comparison p = baseline p (versus ≠)
Calculating power for baseline p = 0.6
α = 0.05

              Sample  Target
Comparison p    Size   Power  Actual Power
         0.5     388     0.8      0.800672

The sample size is for each group.

enter image description here

  • $\begingroup$ Thanks for your response. Good to know I wasn't way off in my thinking. $\endgroup$ – user279473 Apr 2 '20 at 13:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.