1
$\begingroup$

Let's assume I have two small vectors sampled from a large population of 1 Mil trials. One with 50 trials, another with 100 trials. The population probability of success is 0.2, but my estimates from these two vectors are different (let's say 0.05 and 0.1) and therefore the ratio between the two, which should ideally be 1, turns out to be 2. Now, is it possible to take into account the no. of trials in the 2 sets to get a better estimate of the final ratio?

I came across the literature around methods to get confidence intervals of binomial proportions, but what I really want is my ratios to get as close as possible to the ratio of population probabilities. Or, if that's not possible, maybe get the variance around these ratios?

$\endgroup$
1
  • $\begingroup$ Your sub-samples are too small to distinguish between 5% and 10% successes. $\endgroup$
    – BruceET
    Jun 4 at 17:03
1
$\begingroup$

Suppose one of your randomly chosen sub-samples had $x_1 = 2$ successes in $n_1 = 50$ trials and the other had $x_2 = 10$ successes in $n_2 = 100$ trials.

A chi-squared test on a $2\times 2$ contingency table of successes and failures in the two samples would be a traditional approach to compare the two proportions of successes, but your counts are too small for that test to work well. So you should use Fisher's Exact Test, which uses an exact hypergeometric distribution to get its P-value (instead of an approximate chi-squared distribution).

At the 5% level, there is not a significant difference between $\hat p_1 = 0.04$ and $\hat p_2 = 0.10.$ The P-value of the test is $0.3385 > 0.05 = 5\%.$

TBL = rbind(c(2, 10), c(48, 90));  TBL
     [,1] [,2]
[1,]    2   10
[2,]   48   90

fisher.test(TBL)

        Fisher's Exact Test for Count Data

data:  TBL
p-value = 0.3385
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
 0.03866523 1.87501086
sample estimates:
odds ratio 
 0.3770642 

If you tried to use a chi-squared test, you would get a warning message because of the small counts, as shown below:

chisq.test(TBL)

        Pearson's Chi-squared test with Yates' continuity correction

data:  TBL
X-squared = 0.91712, df = 1, p-value = 0.3382

Warning message:
In chisq.test(TBL) : Chi-squared approximation may be incorrect

However, as implemented in R, chisq.test can simulate a more accurate P-value, but again not small enough to reject the null hypothesis that the two proportions are equal.

chisq.test(TBL, sim=T)

        Pearson's Chi-squared test 
        with simulated p-value 
        (based on 2000 replicates)

data:  TBL
X-squared = 1.6304, df = NA, p-value = 0.3418

Note: If you had larger subsamples with about the same proportions of success, then you might see a highly significant result:

TBL2
     [,1] [,2]
[1,]   25  100
[2,]  475  900

chisq.test(TBL2)$p.val
[1] 0.001356245

A 'power and sample size' procedure in Minitab assumes sub-sample of equal sizes, but it might give you an idea what sizes are necessary to be reasonably (90%) sure to detect a difference between the proportions of successes you mention: about $n_1=n_2 = 378.$

Power and Sample Size 

Test for Two Proportions

Testing comparison p = baseline p (versus ≠)
Calculating power for baseline p = 0.1
α = 0.05


              Sample  Target
Comparison p    Size   Power  Actual Power
        0.04     378     0.9      0.900048

The sample size is for each group.

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.