Testing the significance of a difference between a sub-group and the total sample for a binomial distribution

Question

Lets say I have a large group, which include a Bernoulli trial. Summing up the successes (x) and trials (n), I can get an estimate for the average event rate for the population:

# This is R code:
library(Misc)
total_group_n = 5000
total_group_x = 500
binconf(total_group_x, total_group_n)
#> PointEst      Lower    Upper
#>      0.1 0.09198918 0.108625

And for a sub-group in the larger group:

sub_group_n = 1000
sub_group_x = 200
binconf(sub_group_x, sub_group_n)
#> PointEst     Lower    Upper
#>      0.2 0.1763771 0.225919

Here is the question:

What is an appropriate test to conclude that the average event rate for the sub-group is significantly different from that of the total group?

Currently my strategy is to compare the PointEst of p for the subgroup to the confidence intervals for p for the total group.

Is this correct?

Should I be taking both the total-group, and sub-group confidence intervals into account at the same time?

Is there a better way of doing this?

Answers using packages in R (or base R) would be much appreciated.

Answer 1

I assume that the $2 \times 2$ frequency table with marginal totals is as follows:

        In Sub   Out     Total
------------------------------
Succ       200   300       500
Fail       800  3700      4500
------------------------------
Total     1000  4000      5000

It is better to compare those in the subgroup with those outside the subgroup than to compare those in the subgroup with everyone in the total group. (The latter comparison counts those in the subgroup twice and so muddies the comparison. Also see links in Comments by @Glen_b.)

In R, the function prop.test tests the null hypothesis that those in and out of the subgroup have the same Success rate (against the alternative that Success rates differ). The null hypothesis is rejected at any reasonable level of significance, with a very tiny P-value.

prop.test(cbind(c(200,800),c(300,3700)))

        2-sample test for equality of proportions with continuity correction

data:  cbind(c(200, 800), c(300, 3700))
X-squared = 137.5, df = 1, p-value < 2.2e-16
alternative hypothesis: two.sided
95 percent confidence interval:
 0.1767413 0.2677032
sample estimates:
   prop 1    prop 2 
0.4000000 0.1777778 

The function binom.test provides 95% confidence intervals. [The tests themselves are not directly relevant; I have not installed the procedure binconf on my computer.] The CIs for those in and out of the subgroup are $(0.176, 0.226)$ and $(0.067, 0.084),$ respectively. These intervals are quite far from overlapping, so it seems reasonable to conclude that the success rates differ.

binom.test(200, 1000)

        Exact binomial test

data:  200 and 1000
number of successes = 200, number of trials = 1000, p-value < 2.2e-16
alternative hypothesis: true probability of success is not equal to 0.5
95 percent confidence interval:
 0.1756206 0.2261594
sample estimates:
probability of success 
                   0.2 

.

binom.test(300, 4000)

        Exact binomial test

data:  300 and 4000
number of successes = 300, number of trials = 4000, p-value < 2.2e-16
alternative hypothesis: true probability of success is not equal to 0.5
95 percent confidence interval:
 0.06702528 0.08360192
sample estimates:
probability of success 
                 0.075 

Because each of the intervals above has a 5% chance of not covering the their true Success probabilities, comparing them is roughly equivalent to a test at the 10% level.

To conclude a difference in Success probabilities at the 5% level of significance it would be better to use the Bonferroni inequality and compare two CIs at 97.5% level.

The code binom.test(200, 1000, conf.lev = .975) returns the interval $(0.172, 0.230),$ rounded to three places, and binom.test(300, 4000, conf.lev = .975) returns the interval $(0.066, 0.085).$ Again, no overlap.

Notes: (a) I believe that the R procedure binconf gives Wald CIs, now deprecated, while the procedure binom.test gives Agresti-Coull CIs (also called 'plus-4' intervals). Agresti-Coull CIs have been shown to have more stable coverage probabilities than Wald intervals, especially for small $n;$ for the current data the difference is negligible. Perhaps see this Reference and its references.

(b) Procedures prop.test (shown above), chisq.test, and fisher.test (all with with similar syntax) could all be used to test whether the Success probability is the same in and out of the subgroup. Fisher's Exact Test uses the hypergeometric distribution. The other two use normal and chi-squared approximations and typically give very similar P-values.