This question may have been asked before, but I couldn't find it. So, here goes.
From about 3000 data points that can be characterized as "wins" or "losses" (binomial), it turns out that there are 52.8% dumb luck wins. This is my dependent variable.
I also have some additional data that may help in predicting the above wins that could be considered an independent variable.
The question I'm trying to answer is:
If my independent variable can be used to predict 55% wins, how many trials are required (giving me 55% wins) for me to be 99% sure that this wasn't dumb luck?
The following R code is purposely hacky so I can see everything that is happening.
#Run through a set of trial sizes numtri <- seq(2720, 2840, 1) #For a 52.8% probability of dumb luck wins, and the current trial size, #calculate the number of wins at the 99% limit numwin <- qbinom(p=0.99, size=numtri, prob=0.528) #Divide the number of wins at the 99% limit by the trial size to #get the percent wins. perwin <- numwin/numtri #Plot the percent wins versus the trial size, looking for the #"predicted" 55% value. See the plot below. plot(numtri, perwin, type="b", main="Looking for 0.55") grid() #Draw a red line at the 55% level to show which trial sizes are correct abline(h=0.55, lwd=2, col="red") #More than one answer? Why?........Integer issues head(numtri) head(numwin) head(perwin)
From the graph, the answer is: 2740 <= numtri <= 2820
As you can guess, I'm also looking for the required number of trials for 56% wins, 57% wins, 58% wins, etc. So, I'll be automating the process.
Back to my question. Does anyone see a problem with the code, and if not, does anyone have a way to cleanly sneak up on the right and left edges of the "answer"?
Edit (02/13/2011) ==================================================
Per whuber's answer below, I now realize that to compare my alternate 55% wins (which came from the data) with my 52.8% "dumb luck" null (which also came from the data), I have to deal with the fuzz from both measured values. In other words, to be "99% sure" (whatever that means), the 1% tail of BOTH proportions needs to be compared.
For me to get comfortable with whuber's formula for N, I had to reproduce his algebra. In addition, because I know that I'll use this framework for more complicated problems, I bootstrapped the calculations.
Below are some of the results. The upper left graph was produced during the bootstrap process where N (the number of trials) was at 9000. As you can see, the 1% tail for the null extends further into the alternate than the 1% for the alternate. The conclusion? 9000 trials is not enough. The lower left graph is also during the bootstrap process where N was at 13,000. In this case, the 1% tail for the null falls into an area of the alternate that is less than the required 1% value. The conclusion? 13,000 trials is more than enough (too many) for the 1% level.
The upper right graph is where N=11109 (whuber's calculated value) and the 1% tail of the null extends the right amount into the alternate, aligning the 1% tails. The conclusion? 11109 trials are required for the 1% significance level (I am now comfortable with whuber's formula).
The lower right graph is where I used whuber's formula for N and varied the alternate q (at both the 1% and 5% significance levels) so I could have a reference of what might be an "acceptable zone" for alternates. In addition, I overlaid some actual alternates versus their associated number of data points (the blue dots). One point falls well below the 5% level, and even though it might provide 72% "wins", there simply aren't enough data points to differentiate it from 52.8% "dumb luck". So, I'll probably drop that point and the two others below 5% (and the independent variables that were used to select those points).