3
$\begingroup$

I have two small samples denoting if something occured or not. For example:

$s_1=\left(1,1,1,0,0,1\right)$

$s_2=\left(1,0,0,0,0,1,1,0\right)$

I want to test if proportions of occuring $1$ are the same. I cannot use proportion test for two samples, as samples are too small. Can I use Chi square homogenity test and get the robust outcome?

Improve this question
$\endgroup$
3
  • 5
    $\begingroup$ Of course you can use a test of proportions on these data. Your real problem is that you haven't enough data to discern any but the very largest differences between proportions. Unfortunately--in this case--that problem will not go away by choosing another test. You have only three kinds of options: (1) make some strong assumptions (such as a prior distribution); (2) collect more data; (3) accept the results of the test of proportions (which would be not to reject the null hypothesis of equal proportions). Which of these options are ones you would consider? $\endgroup$
    – whuber ♦
    Commented Jan 2, 2015 at 15:38
  • 1
    $\begingroup$ Take a look at fischers exact test. Also, you can find a calculator on the net to do the calculations $\endgroup$
    – user64985
    Commented Jan 6, 2015 at 8:09
  • $\begingroup$ Fishers exact test (or indeed any of the other myriad alternatives) will simply not solve the problem that there's no power to reject any but extremely large differences in proportion. $\endgroup$
    – Glen_b
    Commented Feb 12, 2023 at 21:47

1 Answer 1

Reset to default
3
$\begingroup$

The reason your sample size is a problem for a chi-square test or a test of proportions that relies on approximation to a Normal distribution is that these tests use asymptotic results. In other words, they are built on the assumption that you have "large enough" sample size.

A test that is similar to the chi-square test, but does not rely on asymptotic assumptions, is Fisher's exact test described here: http://en.wikipedia.org/wiki/Fisher%27s_exact_test

Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ Unfortunately, exact implementations of tests of proportions will not make these data appear significantly different (at least for any reasonably small test size). For instance, even an exact one-sided test has a p-value of $\sum_{i=4}^{4+3}\binom{6}{i}\binom{8}{4+3-i}/\binom{6+8}{4+3}=1016/3432\approx 30\%$. $\endgroup$
    – whuber ♦
    Commented Jan 2, 2015 at 16:26
  • 2
    $\begingroup$ Agreed - but it does allow you to perform a test. It is conceivable that there is enough evidence to reject a null hypothesis of equal proportions with the sample sizes Misery gives. $\endgroup$
    – TrynnaDoStat
    Commented Jan 2, 2015 at 16:59
  • 4
    $\begingroup$ No, that's exactly my point: there is not enough evidence here. To obtain a significant result (at $\alpha=0.05$) with two samples of sizes six and eight, conditional on observing seven positives, you would need all elements of the smaller sample to be positive. And you could determine that using the standard test of proportions (but, as you point out, only by applying the exact sampling distribution, because an asymptotic approximation would be suspect). $\endgroup$
    – whuber ♦
    Commented Jan 2, 2015 at 17:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.