How to calculate the number of trials for a 'meaningful' result Apologies for newbie question but have failed to understand what I've read about in this area so pointers to the correct methods would focus my attention.
I have a piece of software which when tested repeatedly in the same way fails occasionally - say once in 20 runs.
If I make a change to fix the problem, how many times should I test it to be confident (say 95%) that the changed software is actually better than the original.
More generally: if a test fails a out of b times. After the fix, how many times (c) must it run without any failures to be confident to d%.
 A: A rule of thumb says that if you run n trials and see no events, then the 95% confidence interval for the rate of the events is from 0 to 3/n.  So if you want to be 95% confident that the true proportion of failures is below p, then do 3/p runs and if none fail, you can be 95% confident that the proportion is below p.
A: You have to guess what the new failure rate is going to be (maybe based on a pilot study) and then calculate the chance of type II error.
There are formulae for this, but I sometimes find it easier and more convincing to simulate the results. Here's some sloppy R code that will use simulations to estimate the power. (The link is for a paired t-test, and I simulated a two-sample t-test, so you'd have to adjust one of these for them to be equivalent.)
#Compare the failure rates of two random samples
fail <- function(n,p) rbinom(n,1,p)
#It would be good to change t.test to a more powerful test (and see how that affects power).
compare <- function (n,p.old=1/20,p.new=1/10) t.test(fail(n,p.old),fail(n,p.new))$p.value

#Use 50 simulations. Increase this for higher precision
simulations<-50

#Parametrize sample size
samplesize<-seq(50,1000,10)
#What's the 95%ile p-value of the simulations
q95_p.value<-function(samplesize) quantile(replicate(simulations,compare(samplesize)),0.95)

#Plot the p-values
plot(samplesize,sapply(samplesize,q95_p.value),ylim=0:1,ylab='95%ile simulated p-value')
#If the 95%ile p-value for the sample size is below the line, then power is probably above 95%
abline(h=0.05)

Here is the resulting plot assuming that the old failure rate is $\frac{1}{20}$ and the new $\frac{1}{10}$. If the 95%ile p-value for the sample size is below the line, then power is probably above 95%.

There's probably an R package that does this better.
And I must say that those sample sizes look awfully large. Maybe I made a mistake somewhere.
A: I believe the answer you are looking for is
Number of trials = log(1-CF)/log(1-failure rate),
where CF is the confidence factor you are looking for.
For your example, where you want a confidence of 95% that you solved a problem that is expected to occur 5% of the time, it would be
Number of trials = log(1-.95)/log(1-.05) = 59 trials rounding up.
