Estimate the proportion and variance in a simple binomial cluster design

Question

I am trying to estimate a binomial proportion p, say, from a sample of binomials. There are k subjects. Associated with each subject is a sample size $n_i$ and a count $x_i$ of items, where $x_i$ is distributed as a binomial $(p, n_i)$. I can assume that the sample sizes are not a function of $p$.

In sampling terms, the sampling unit is the subject, not the number of items $x_i$.

I want to estimate p and provide a confidence interval for it.

Should I take $$\hat{p}=\frac{\sum_i x_i}{\sum_i n_i}$$ or should I take the average of the cluster means? $$ \frac{1}{k} \sum_i \frac{x_i}{n_i}$$

And how should I estimate the standard deviation?

Answer 1

While the first estimate is the MLE under perfect conditions, I believe there's enough scope for both estimating the proportion and testing the assumption that really all proportions are the same. You can do this as a nested logistic regression with just a common intercept vs. a bunch of fixed effects for each subject. Or a variance component in a GLMM way. The test that all coefficients are equal to zero (or that the variance component is equal to zero) would justify the first approach. If this test fails, you have to admit that your success probabilities vary between subjects; in this case, the overall probability may still be a valid population parameter and a target of inference, but only the second formula is applicable.

The variance estimator is formula (2.3-8) in Korn & Graubard 1999 (which is a great book worth having if you work with surveys to any appreciable extent).

Answer 2

You present a very çlean situation, but the real world is often not really clean. Are the k subjects really fungible? Are the sample sizes a function of the subjects? In any case, the average of the p values for each subject would be less influenced by the subjects with larger n's. That might be a conservative approach. You might calculate the p's both ways and see if they differ. If they differ, you might try to find out why. If the situation is as clean as you describe, the first approach would be best, since it uses all the information. In any case, the SD of p = sqrt (pq/n) if you are sampling from an infinite binomial distribution.

Answer 3

I agree that the first estimate is the best. Each cluster represents an independent set of iid Bernoulli random variables. So pooling them gives you a single binomial with N equal to the sum of the n$_i$s and estimate 1 is the mle for p. The variance estimate for p^ is p^(1-p^)/N where p^= ∑$_i$ x$_i$/ ∑$_i$ n$_i$. N=∑$_i$ n$_i$.