Is there a way to reduce a sample size such that I gain roughly the same amount of information out of it as I would get from a bigger one? I was thinking of Cochran's formula for determining the ideal sample size under maximum variability, however, I couldn't find a way to formulate my problem such that I end up with some sort of relevant attribute.
Nonetheless, suppose I want to have 99.9% accuracy for a given task –– this means 1 fail out of 1000 experiments. However, performing 1000 experiments is quite expensive, so I want to reduce that number. Is there a way, for example, to perform 100 experiments and use that data to talk "as if" I would have done 1000?
Any hints would be appreciated!
 A: I'm assuming you want to confirm your 99.9% accuracy based on experiments. But actually, running 1000 experiments won't confirm a 99.9% succes rate. You may be unlucky, and have 2 fails, even if the "true" success rate is 99.9%. There are various ways to arrive at a confidence interval for the succes rate, based on data, and there are also ways to figure out how many experiments you need for getting a small enough (for your purposes) confidence interval. As you can imagine, if you want to be sure the "true" success rate is between 99.45% and 99.55%, you're going to need to run many more than 1000 experiments.
A: If you have very low proportions, approximations based on the normal distribution like the Wilson Interval do not have a good coverage probability.
You should thus use the Clopper-Pearson Interval, which, e.g. the R function binom.test computes:
> ci <- binom.test(0, 100, conf.level=0.95)$conf.int
> ci[1:2]
[1] 0.00000000 0.03621669

For zero hits, the interval can even be computed in closed form as $[0,1-\sqrt[n]{\alpha/2}]$, where $\alpha$ is the error probability (i.e. $1-\alpha$ is the confidence level). For other values, however, it must be computed numerically.
See section 3.1. of this Technical Report for the underlying ideas and formulas. It also shows the problems of other confidence intervals for small (or large) $p$.
