# H to check if proportions in two small samples are the same

I have two small samples denoting if something occured or not. For example:

$s_1=\left(1,1,1,0,0,1\right)$

$s_2=\left(1,0,0,0,0,1,1,0\right)$

I want to test if proportions of occuring $1$ are the same. I cannot use proportion test for two samples, as samples are too small. Can I use Chi square homogenity test and get the robust outcome?

• Of course you can use a test of proportions on these data. Your real problem is that you haven't enough data to discern any but the very largest differences between proportions. Unfortunately--in this case--that problem will not go away by choosing another test. You have only three kinds of options: (1) make some strong assumptions (such as a prior distribution); (2) collect more data; (3) accept the results of the test of proportions (which would be not to reject the null hypothesis of equal proportions). Which of these options are ones you would consider?
– whuber
Commented Jan 2, 2015 at 15:38
• Take a look at fischers exact test. Also, you can find a calculator on the net to do the calculations
– user64985
Commented Jan 6, 2015 at 8:09
• Fishers exact test (or indeed any of the other myriad alternatives) will simply not solve the problem that there's no power to reject any but extremely large differences in proportion. Commented Feb 12, 2023 at 21:47

• Unfortunately, exact implementations of tests of proportions will not make these data appear significantly different (at least for any reasonably small test size). For instance, even an exact one-sided test has a p-value of $\sum_{i=4}^{4+3}\binom{6}{i}\binom{8}{4+3-i}/\binom{6+8}{4+3}=1016/3432\approx 30\%$.
• No, that's exactly my point: there is not enough evidence here. To obtain a significant result (at $\alpha=0.05$) with two samples of sizes six and eight, conditional on observing seven positives, you would need all elements of the smaller sample to be positive. And you could determine that using the standard test of proportions (but, as you point out, only by applying the exact sampling distribution, because an asymptotic approximation would be suspect).