Testing a binomial model I have a computational experiment that takes a size parameter $s$ and returns true or false.
It has been suggested that the result of the experiment is independent of $s$, but I don't think so.  I expect a general decrease in the numbers, though only logarithmically dependent on $s$.  (This makes it hard to detect and is the essential reason for my posting -- a larger effect would be easier to detect.)
For s = 5..30 I have generated 1000 samples for which I give the number of occurrences, respectively:
108, 108, 99, 99, 98, 87, 82, 85, 84, 94, 97, 87, 86, 102, 82, 84, 83, 87, 75, 69, 66, 79, 91, 81, 74, 74
Another wrinkle: the time needed to generate the samples depends on the size parameter.  Making 1000 tests at size 10 took less than a second; at size 30 it takes nearly a minute.  So I could generate many more examples at small sizes if that would be helpful.  ($s$ relates to $10^s$, so I can't pick too many more than $10^s$ for a given $s$.  But I could run a billion tests at s = 10 without trouble [in a week or two].)
The flip side, of course, is that it's difficult to generate large examples, even though those are the ones that would be most useful.  With a bit more effort (half a dozen processor-hours) I was able to do 1000 tests at higher levels:
s = 40: 75
s = 45: 81
s = 50: 74
s = 55: 73
s = 60: 74
s = 65: 68
So my question: How should I design an experiment to decide if these are identically distributed?  Once I have my data, how can I decide (for some $\alpha$) whether to reject or not reject the hypothesis that the distributions are identical, i.e., independent of $s$?
For example, I could continue to generate data for 1000 tests at each value of $s$ from 31 to 70.  But given the increasing time needed as the parameter increases I'm not sure that this is an appropriate way to sample the data.  But once I introduce non-uniformity I'm not sure what testing technique would be appropriate.  (I'm a statistical neophyte.)
 A: Sometimes, the data is sufficiently strong that you can get by with testing a much weaker assertion than what you are actually interested in. In this case, a very weak consequence of the null hypothesis (the probability is independent of $s$) would be that the cases $5 \leq s \leq 17$ and $18 \leq s \leq 30$ have the same probability of success. You can test this with a two-sample t test; acceptance of the null hypothesis wouldn't tell you very much (only that this weak consequence is acceptable), but rejection of the consequence implies rejection of the original, stronger, null hypothesis. I ran this test in two different ways - one by testing the $2 \times 13$ numbers of successes, and one by testing the $2 \times 13000$ zeroes and ones for the individual outcomes. In both cases, the $p$-value was pretty small (0.0016 and 0.00024, respectively), so for $\alpha > 0.00024$ I believe we can reject the hypothesis.
Of course there are many situations where this type of laziness doesn't work; then we need more advanced analysis to give us a definitive result. But sometimes a lazy approach is good enough.
(P.S. - I later realized that a cheap strengthening of the test would be to make it one-sided; that should improve the values even further with hardly any extra effort.)
A: A good choice to analyze these data would be a generalized linear model for a binomial response and a logit link.  Conceptually it's simple: it says that each test has a chance $f(s)$ of succeeding, so that 1000 runs at a given value of $s$ should behave like 1000 flips of a coin having an $f(s)$ chance of success.  The chances are not expressed as probabilities, but as the logarithms of the odds: that's the logit.  Your supposition is that these log odds are a linear function of $\log(s)$.  The function $f$ is the combination of this logarithm and the logit.
However, you don't really need this level of sophistication because (a) you have run the same number of tests each time, (b) the number is large enough that the response counts are not small, and (c) the expected changes in response rate and its variance for this range of $s$ (from 5 through 70) are small.  Thus you will get essentially the same results if you do ordinary least squares regression of the logit of response/1000 against $\log(s)$.  This is simple enough even to do with a spreadsheet.

This plot shows the re-expressed data and their least squares fit.  It's a good fit; the residuals tend to be evenly scattered around it; and no points are extremely far to the right or the left.  There is a slight tendency for residuals to be positive for both small and large values of $\log(s)$, hinting at a quadratic term in $\log(s)$.  If that's a possibility, run more experiments for $s \le 10$ and $s \ge 50$.
The OLS table is
           y |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         lns |   -.173364   .0238979    -7.25   0.000      -.22217   -.1245579
       _cons |  -1.874851   .0727821   -25.76   0.000    -2.023492    -1.72621
------------------------------------------------------------------------------

and, for comparison, the (more accurate) GLM table is
    response |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         lns |  -.1758287   .0305201    -5.76   0.000    -.2356469   -.1160104
       _cons |  -1.864569    .091082   -20.47   0.000    -2.043086   -1.686051
------------------------------------------------------------------------------

The closeness of the coefficients show that both techniques are fitting the same curve.  The logarithmic term is highly significant for both.  OLS is a little overconfident (it gets more extreme p values), but that doesn't matter here.
As a final check, we can back-transform the fit.  I have placed two bands at one standard deviation (as predicted by the fit at each point: the bands actually get narrower as $s$ gets larger).  If anything, the scatter is too tight: we would expect one-third or so of these 32 points, or 10 of them, to lie beyond the bands, but only 5 do.  (I wouldn't read much into that, but I would be worried if the opposite were true: if the majority of points fell outside these bands, I would worry about the accuracy of the binomial model.)

If you have the capability of running a GLM, then you can use it to analyze a dataset in which the numbers of tests per value of $s$ vary.  Because you already have about six times as much data as you need (to show the log trend is significant), I would recommend substantially reducing the sizes of future experiments, perhaps down to 100 or even less, rather than 1000 at a time.  You certainly can optimize your experimentation to get the most additional information for the extra work, but if you're almost done, it's less work overall just to continue with these rough heuristics.
A: This is a neat design of experiments exercise. You have a regression model, something like ${\rm Prob}[{\rm test}=1|s] = a + b \log s + \epsilon$, and you want to estimate the parameter $b$ subject to computational costs. This is akin to Neymann-Chuprow allocation in survey statistics. As a first approximation, you can utilize Neymann formula with $h$ being each individual value of $s$ and $C(s)$ being the time it takes to produce a single test. Basically, you'd want to take the number of tests that is inversely proportional to $\sqrt{C(s)}$. As a more accurate approximation, you can develop a model for the cost as a function of $s$, and plug it into the variance expression for the coefficient $b$ (the corresponding entry in the standard $\sigma^2 (X'X)^{-1}$ formula assuming the variances are constant... which is not exactly right, but for a small effect, is not a terrible approximation).
Update: per whuber's suggestion in the comments section: it might be worth checking the assumptions of your model. As long as we are in an agreement that this is an experimental design situation, the (rather small) subfield of the expertimental design literature that deals with deviations from the function form is called "robust design" literature. If you knew you had a linear trend, the best design would be to collect the data in the two extreme points ($s=5$ and $s=30$). However, to test for deviations from linearity, you would need to use a design that spreads the points across the whole domain (in the simple case of constant costs and constant variance, uniformly between the two extremes). For you, this basically means an extended analysis of the (existing or to be collected) data, in which you would run say a regression $\mbox{Prob}[\mbox{test}=1|s] = b_0 + b_1 s + b_2 s^2 + b_3 s^3 + \epsilon$ to see if you have higher order wiggles in your line (or add your favorite $\log s$ term there if you insist on the log dependence).
