# Taking a random factor into account while measuring a proportion a low N

I need to be neat in measuring the success rate of a treatment. It is anyway pretty high. But as it is all about ecology, multpliying experiments is difficult.

I have treated $N = 20$ individuals, $18$ succeded. This is a $\tau = .9$ success rate.

I used the Jeffrey's table to get the uncertainty about this rate (Brown 2013), which seems then to be $CI_{95\%} = [.716,.979]$.

Now, to be really neat, I'd like to increase it a little, taking into account the fact that, in reality, $N = N_1 + N_2 + N_3 + N_4$ with all $N_i = 5$, for the treatments have been given at 4 different dates.

How do I take this replication into account ? How do I integrate a random factor to a proportion estimation ?

I am aware it might be too much for not much, and that the raw data for $N = 20$ will be pretty convincing anyway. :)

[EDIT:]

A more narrow question seems to arise from your comments : How do I estimate, in the first place, whether or not the date $i$ is likely to have an impact on the success rate? Put it another way: How do I assess the null hypothesis $H_0: \tau_1 = \tau_2 = \dots$ with this (rather poor) data?

• If this hypothesis was to be validated by the data, then I would use this $CI_{95\%}$ and everthing would be fine.
• If it was to be invalidated by the data then How much wider should it be? (it would then represent the confidence interval of an "overall success rate $\tau$" (?))

(once again, GLMs do seem to be the immediate answer but I'm far below their asymptotic assumptions..)

(and once again, at the end of the day, $N$ might actually turn out to be so low that doing statistics is not a relevant option anymore ;)

• What's your model? Is it that the population proportion (i.e. the underlying probability) is the same for each small sample, or that they're actually samples representing different populations with some potential variation between them in probability? Jul 28 '14 at 0:44
• Tough question! It'd be possible to fit some sort of GLMM with a random day effect and just the intercept as the fixed effect. But all the results would be based on asymptotics, and you are nowhere near an asymptote! And leaving it as-is is too optimistic, as it assumes a constant probability of success for all 20 trials. (And that they are independent -- are they, or are the 5 trials per day systematic in some way?) Jul 28 '14 at 1:01
• @Glen_b: I do strongly assume that the underlying probability is the same for each $N_i$ sample, for the only difference between them is the date of the replicate (same population then, unless you consider the "date" as a population characteristic (and you might be right, for who knows date is weather, moon, etc.. anyway sth random)). If there was a null hypothesis saying $H_0:$ « the date has no effect on $\tau$ », then I would wish it to be (likely to be) true. But if it turned out to be (likely to be) wrong, I would wish the effect of my treatment to be significantly high at any date. Jul 28 '14 at 1:43
• @RussLenth: Yes they are independent. On each date $i$, $N_i$ individuals were randomly drawn from the population then treated. How expensive (I mean « how much too-optimistic ») would it be to simply assume that there is no date effect and that my $CI_{95\%}$ is valid? I guess this would directly depend on the variation in $\tau_i$.. but I also guess retrieving all the $\tau_i$ then abruptly measure their dispersion isn't a good idea either.. ;) Jul 28 '14 at 3:11
• Don't confuse true $\tau_i$ with the estimated $\hat\tau_i$. If the true $\tau_i$ can be assumed the same from day to day, then you're fine. Else, the more they vary, the more optimistic your interval is. (By this I mean the interval is too narrow.) Jul 28 '14 at 3:18

If your strong belief is (as you say) that the probability is constant across samples and there's independence within-sample, you simply combine all of them into one big sample of $N_1+N_2+...$ and proceed as before. This is very easy.

If on the other hand there is likely to be an amount of variation (not simply due to random variation, I mean variation in the underlying proportion itself) from a variety of subpopulations about an overall population proportion, you could consider trying a mixed model (and because it's binomial data, it would be a generalized linear mixed model), where the random factor is the group (the different times).

In response to questions in comments:

With very small samples (n=3 or 4 say) there's probably little to do but assume.

With slightly larger samples you have a couple of possibilities for measuring the extra-variation that this would produce:

i) One possibility is to look at the homogeneity across samples. You could do a formal chi-squared test of heterogeneity, perhaps with an exact test or even just simulated p-values if too many of the expected values are small for the chi-square approximation to work.

However, note that formal hypothesis testing of assumptions isn't answering the right question.

The question of interest with regard to assumptions isn't 'is the assumption false by enough that we can reject it?' -- with large enough sample size you always will reject, so given the assumption is probably always strictly false, that's merely telling you about your sample size. The real question is 'is it so false that it badly impacts the original inference?', which is an effect size question). This leads us to:

ii) You could simply consider the differences visually by looking at/plotting the proportions and deciding if they're reasonably homogenous. You could make some choice about what you considered a big difference (in terms of 'does it matter' rather than 'is it statistically significant'). If you wished, you could go further by calculating the standard error of the proportions for each group. For example if the two standard errors of the proportions are $s$ and $l$ respectively (where $s<l$), then $1.96\sqrt{s^2+l^2}\leq 1.96(l+\frac{s}{1+\sqrt{2}})<2l+s$, so if you mark $\pm 1$ and $\pm 2$ standard errors either side of each proportion, a rough-but-conservative* 5% two-sample interval for the difference in two proportions would not include 0 if neither of the wider intervals overlapped the narrow interval on the other group. You'd expect a bit less than 5% of such pairwise interval-comparisons to fail to show any overlap, so if there were many such failures-to-overlap which were also important in size, you might find the assumption of homogeneity neither tenable nor a useful approximation.

* you could come closer if you did them at 1.96 and 0.82 standard errors for the outer and inner intervals, but there's already several levels of approximation here anyway.

Discussion of the derivation of (and more details of the use of) the standard error bars and how to visually judge that two proportions would be different at the 5% level.

Refer to the Wikipedia page on binomial proportion confidence intervals for various intervals and references, but I give some additional ones here.

In this case I'll write as if the asymptotic standard errors apply, but just as for single samples you're probably better off using something like Agresti-Coull type intervals -- say the bounds derived by using the "add two successes and two failures" rule $\tilde{p}=\frac{X+2}{\tilde{n}}=\frac{X+2}{n+4}$ estimate with the associated standard errors $s(\tilde{p})=\sqrt{\tilde{p}(1-\tilde{p})/\tilde{n}}$ (i.e. generate the usual simple symmetric Wald interval but about a biased estimate, which has reasonably good coverage properties).

However, we're doing a comparison of two proportions. In that case (see Agresti and Caffo 1) we also add 4 observations, one success and one failure to each of the two samples, so now $\tilde{p}_i=\frac{X_i+1}{\tilde{n}_i}=\frac{X_i+1}{n_i+2}$ with standard error $s(\tilde{p})=\sqrt{\tilde{p}(1-\tilde{p})/\tilde{n}}$.

The advantage of this over some of the other possible intervals (one it shares with the Wald) is that the Agresti-Caffo approach translates relatively easily to a visual display (and isn't so hard to explain). We're not so concerned about perfect coverage, so the Agresti-Caffo should be sufficient.

Indeed, if you wanted to just use the asymptotic $\hat{p}=X/n$ with standard error $s(\hat{p})=\sqrt{\hat{p}(1-\hat{p})/n}$ that might do okay for what is largely intended to be a visual sense of whether the differences are important.

In any case, assume for group $i$, we have some estimate, $\tilde{p}_i$ and some standard error $s_i(\tilde{p})$.

Then when comparing two such estimates, using a Wald type interval for the difference, we could compute the standard error of the difference as $s_d = \sqrt{s_1^2+s_2^2}$. This is convenient but not so useful when looking for a visual difference.

Now come some approximations.

Let $s=\min(s_1,s_2)$ and $l=\max(s_1,s_2)$.

Then $s_d = \sqrt{s^2+l^2}=l\sqrt{r^2+1}$ where $r =s/l$ lies between $0$ and $1$.

We can approximate $\sqrt{r^2+1}$ by the upper bound $1+\frac{r}{1+\sqrt{2}}$.

Hence

$s_d = l\sqrt{r^2+1} \leq l\ (1+\frac{r}{1+\sqrt{2}}) =l+\frac{s}{1+\sqrt{2}}$

Now $1.96 s_d \leq 1.96\ l\ +\ s\frac{1.96}{1+\sqrt{2}}=1.96\ l\ +\ 0.82\ s \leq 2\ l + s$.

So if the proportions are separated by at least twice the larger of the standard errors plus the smaller, the pairwise $(1-\alpha)$ interval for the difference would not include 0.

Hence if you put intervals around the estimates at $\pm 1 s_i$ and $\pm 2 s_i$, and if neither of the wider intervals overlaps the narrower one in the other group, an interval for the difference in the original proportions on which the calculations are based would not include 0.

In the above diagram, neither of the wider intervals overlaps the opposite narrower interval, which guarantees that $\tilde{p}_1-\tilde{p}_2$ is larger than $2l+s$ (note that we don't need to be able to tell which is $l$ and which is $s$, so 0 is not included in the 95% CI for the difference.

A less conservative interval for the difference would use the inner bars at $\pm 0.82 s_i$ and the outer ones at $\pm 1.96 s_i$.

But, just to repeat my earlier discussion, practical importance needs to be taken into account. In large samples this will tell you about trivial differences. While it can make sense to use it in very small samples (since it may tell you that differences that seem large may still not be from different populations), it's basically then just a test of homogeneity done in a way that doesn't control overall type I error, so you you should keep that in mind.

Since I derived the approach for this question, there's no reference I can offer for it, but I wouldn't be at all surprised if it - or something quite similar - hadn't already been done.

1: Agresti, A., and Caffo, B.(2000),
"Simple and effective confidence intervals for proportions and difference of proportions result from adding two successes and two failures."
The American Statistician, 54 (4), 280–288.

• This does make sense.. But how can I measure this eventuality from my data? (this eventuality: « there is likely to be an amount of variation from a variety of subpopulations about an overall population proportion »). And also, as RussLength noticed, the GLMM « would be based on asymptotics, and [I am] nowhere near a asymptote!».. °~° Jul 28 '14 at 3:20
• I've added some discussion to my answer Jul 28 '14 at 5:23
• `Totally agreed with ii), for this is what I had in mind with "it might be too much for not much". My final goal is just to show that the data suggests a good treatment efficiency -- no matter the date it is done --, and the raw visual proportions are just pretty convincing in this way. Thank you for this common sense and for this formula that will even allow me to get error bars. (for anyone knows reviewers often feel lost without'em, no matter how low your actual $N$ might be ;) Edit: by the way, would you have more information about this formula and how you got there? (any good reading?) Jul 28 '14 at 6:53
• I just derived it for your question but no doubt it or something like it has been done before. I can include more details of the derivation if you wish. Jul 28 '14 at 8:16
• Amazing. Now I just feel spoiled for choice about which bars to use, and the problem of feeling spoiled for choice is nothing but.. a delightful non-problem. Thanks a lot! ^ ^ Jul 28 '14 at 19:54