Two fair coins, why increasing trials in prop.test not reduce false positive?

Question

I am simulating two fair coins and running a prop.test, how come increasing number of trials doesn't decrease false positive? I'm suspecting it's in some underlying assumptions in the R functions that I'm using which I missed.

This binomial coin flip simulator takes argument N, for number of trials, and returns the p-value of an two-sided prop.test.

test.flips <- function(N, A.prop=0.5, B.prop=0.5) {
  heads.A <- rbinom(1, N, A.prop)
  heads.B <- rbinom(1, N, B.prop)
  test <- prop.test(c(heads.A, heads.B), n=c(N, N), alternative="two.sided")
  return(test$p.value)
}

I do 100 times of 1000 trials each to get this set of p-values, i.e. p.values <- replicate(100, test.flips(1000))

The red horizontal line denotes a 0.05 p-value. Notice that we have a few false positives.

Thus I do a power calculation, power <- power.prop.test(p1=0.5, p2=0.501, power=0.90, alternative="two.sided") to find that N = 5253704 if power=0.90.

So I do this again, p.values <- replicate(100, test.flips(round(power$n))) with N set to that high number of trials.

But the number of false positives didn't improve as shown here:

If all else remains equal (same sig. level), how come I'm not seeing what's gained by increasing sample size? What's wrong and how do I fix this please?

Edit:

• state clearer question in end
• As requested, here's prop.test()$statistic versus sample size.

Answer 1

As explained by Stephan, everything is at it should be. In fact, the issue really has nothing to with prop.test specifically and the most fruitful for you might be to (re)read some introductory material on statistical testing but it's relatively easy to adapt your simulation to illustrate what's going on.

What the power analysis tells you is that, for population proportions .5 and .501, you should find a significant difference 9 times out of 10 if you take samples of 5253704 observations and an error level of 5%. And indeed, this is approximately the results you get if you run the corresponding simulation, namely p.values <- replicate(100, test.flips(round(power$n), A.prop=0.5, B.prop=0.501)). Of course, in this case, significant results aren't “false positives” anymore because .5 really is different from .501.

To reduce the number of false positives, you simply need to adjust the error level when interpreting the test, i.e. to accept only p-values under .01 or .001 as significant. Graphically, this would correspond to lowering the red line in your plots.

Answer 2

There is nothing wrong. Recall the definition of the p value: the probability of erroneously rejecting the null hypothesis (that the coins are fair) when it is true. If you have fair coins and test them against an alpha level of 0.05, you should erroneously reject the (true) null in one out of twenty cases.

The sample size enters only in calculating the p value of the test and will make less and less of a difference as the t distribution converges to a normal distribution.

Your power calculation assumes that the coins are not fair (a power calculation only makes sense if we assume that the null hypothesis is false), so it has nothing to say about the case where the null hypothesis is true.

Answer 3

Having unconfused myself thanks to the helps here. I made new plots to show what I wanted to show originally.

Testing two fair coins. As Gaël pointed out, false-positive (spikes below the red line) does not vary with sample size as it's what the test guarantees (specified by significance level).

Testing two marginally different coins, i.e. 0.50 vs. 0.51 probabilities. False-negative does indeed decrease (above red line) as sample size increase.