Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am simulating two fair coins and running a prop.test, how come increasing number of trials doesn't decrease false positive? I'm suspecting it's in some underlying assumptions in the R functions that I'm using which I missed.

This binomial coin flip simulator takes argument N, for number of trials, and returns the p-value of an two-sided prop.test.

test.flips <- function(N, A.prop=0.5, B.prop=0.5) {
  heads.A <- rbinom(1, N, A.prop)
  heads.B <- rbinom(1, N, B.prop)
  test <- prop.test(c(heads.A, heads.B), n=c(N, N), alternative="two.sided")

I do 100 times of 1000 trials each to get this set of p-values, i.e. p.values <- replicate(100, test.flips(1000))

enter image description here

The red horizontal line denotes a 0.05 p-value. Notice that we have a few false positives.

Thus I do a power calculation, power <- power.prop.test(p1=0.5, p2=0.501, power=0.90, alternative="two.sided") to find that N = 5253704 if power=0.90.

So I do this again, p.values <- replicate(100, test.flips(round(power$n))) with N set to that high number of trials.

But the number of false positives didn't improve as shown here:

enter image description here

If all else remains equal (same sig. level), how come I'm not seeing what's gained by increasing sample size? What's wrong and how do I fix this please?


  • state clearer question in end
  • As requested, here's prop.test()$statistic versus sample size.

enter image description here

share|improve this question
You would need to vary the conf.level argument to prop.test if you want to change the nominal false positive rate (whose default is 95%). – whuber Jan 10 '13 at 15:29
up vote 2 down vote accepted

As explained by Stephan, everything is at it should be. In fact, the issue really has nothing to with prop.test specifically and the most fruitful for you might be to (re)read some introductory material on statistical testing but it's relatively easy to adapt your simulation to illustrate what's going on.

What the power analysis tells you is that, for population proportions .5 and .501, you should find a significant difference 9 times out of 10 if you take samples of 5253704 observations and an error level of 5%. And indeed, this is approximately the results you get if you run the corresponding simulation, namely p.values <- replicate(100, test.flips(round(power$n), A.prop=0.5, B.prop=0.501)). Of course, in this case, significant results aren't “false positives” anymore because .5 really is different from .501.

To reduce the number of false positives, you simply need to adjust the error level when interpreting the test, i.e. to accept only p-values under .01 or .001 as significant. Graphically, this would correspond to lowering the red line in your plots.

share|improve this answer
But that would just be changing the threshold. I simply want to illustrate what's being gained by running more trials. – Paul Lam Jan 10 '13 at 16:53
What's being gained by running more trials is an increase in the number of true positives or, equivalently, a decrease in the number of false negatives. That the number of false positives does not change is precisely the guarantee of the test (provided its assumptions are met). – Gala Jan 10 '13 at 16:55
Thanks! Makes sense now. I've been confusing myself. Now I need to figure out how to illustrate this decreasing false negative... – Paul Lam Jan 10 '13 at 17:02
In addition, running more trials will narrow the confidence interval estimating the difference between the 2 coins. This can be very important when considering practical difference. You can never eliminate the possibility of false positive (unless $\alpha=0$) and you can never guarentee that they proportions are the same, but with enough flips you can be confident that they don't differ by more than some small amount (this is called equivalence). – Greg Snow Jan 10 '13 at 18:58

There is nothing wrong. Recall the definition of the p value: the probability of erroneously rejecting the null hypothesis (that the coins are fair) when it is true. If you have fair coins and test them against an alpha level of 0.05, you should erroneously reject the (true) null in one out of twenty cases.

The sample size enters only in calculating the p value of the test and will make less and less of a difference as the t distribution converges to a normal distribution.

Your power calculation assumes that the coins are not fair (a power calculation only makes sense if we assume that the null hypothesis is false), so it has nothing to say about the case where the null hypothesis is true.

share|improve this answer
prop.test()$statistic gives me a X-sq value. Isn't that proportional to p-value? I edited my post to show test statistic vs. sample size. It doesn't look to be decreasing? – Paul Lam Jan 10 '13 at 16:13
No, the chi-squared value is most certainly not proportional to the p value. And you don't seem to be plotting the statistic against the sample size, but against the replicate (with constant sample size of 100). – Stephan Kolassa Jan 10 '13 at 16:19
Yes, but the O.P. is right about one thing: the (null) distribution of the t-statistic, as the sample size increases, converges to the standard Normal distribution. Your statement that "with increasing sample size, the test statistic for a given p value will decrease" is barely correct, and not in any consequential way. When I first read your answer (and upvoted it), I understood this statement to refer to how the effect size is measured rather than the test statistic per se: a fixed difference in proportions, as an effect size, grows in proportion to the square root of sample size. – whuber Jan 10 '13 at 16:23
Agreed. Sorry. Will edit to avoid possible confusion. – Stephan Kolassa Jan 10 '13 at 16:24
thank you, Stephan, I like your answer too. But Gael's is when my thought clicked so I chose his. I upvoted yours though. – Paul Lam Jan 10 '13 at 17:21

Having unconfused myself thanks to the helps here. I made new plots to show what I wanted to show originally.

Testing two fair coins. As Gaël pointed out, false-positive (spikes below the red line) does not vary with sample size as it's what the test guarantees (specified by significance level).

enter image description here

Testing two marginally different coins, i.e. 0.50 vs. 0.51 probabilities. False-negative does indeed decrease (above red line) as sample size increase.

enter image description here

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.