I have two small samples denoting if something occured or not. For example:

$s_1=\left(1,1,1,0,0,1\right)$

$s_2=\left(1,0,0,0,0,1,1,0\right)$

I want to test if proportions of occuring $1$ are the same. I cannot use proportion test for two samples, as samples are too small. Can I use Chi square homogenity test and get the robust outcome?

  • 5
    Of course you can use a test of proportions on these data. Your real problem is that you haven't enough data to discern any but the very largest differences between proportions. Unfortunately--in this case--that problem will not go away by choosing another test. You have only three kinds of options: (1) make some strong assumptions (such as a prior distribution); (2) collect more data; (3) accept the results of the test of proportions (which would be not to reject the null hypothesis of equal proportions). Which of these options are ones you would consider? – whuber Jan 2 '15 at 15:38
  • 1
    Take a look at fischers exact test. Also, you can find a calculator on the net to do the calculations – user64985 Jan 6 '15 at 8:09
up vote 2 down vote accepted

The reason your sample size is a problem for a chi-square test or a test of proportions that relies on approximation to a Normal distribution is that these tests use asymptotic results. In other words, they are built on the assumption that you have "large enough" sample size.

A test that is similar to the chi-square test, but does not rely on asymptotic assumptions, is Fisher's exact test described here: http://en.wikipedia.org/wiki/Fisher%27s_exact_test

  • 2
    Unfortunately, exact implementations of tests of proportions will not make these data appear significantly different (at least for any reasonably small test size). For instance, even an exact one-sided test has a p-value of $\sum_{i=4}^{4+3}\binom{6}{i}\binom{8}{4+3-i}/\binom{6+8}{4+3}=1016/3432\approx 30\%$. – whuber Jan 2 '15 at 16:26
  • 2
    Agreed - but it does allow you to perform a test. It is conceivable that there is enough evidence to reject a null hypothesis of equal proportions with the sample sizes Misery gives. – TrynnaDoStat Jan 2 '15 at 16:59
  • 3
    No, that's exactly my point: there is not enough evidence here. To obtain a significant result (at $\alpha=0.05$) with two samples of sizes six and eight, conditional on observing seven positives, you would need all elements of the smaller sample to be positive. And you could determine that using the standard test of proportions (but, as you point out, only by applying the exact sampling distribution, because an asymptotic approximation would be suspect). – whuber Jan 2 '15 at 17:02

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.