I've run a test with one control and one experiment group, and am questioning myself on whether or not I've used the right test (or if significance can even be calculated on the following sample sizes).

The data is as follows:

The control cohort (A) had 63 people see the treatment and 1 person performed the action (1.59%)

The Experiment cohort (B) had 64 people see the treatment and 9 people performed the action (14.1%)

I used a z-test for two population proportions (this equation: http://www.socscistatistics.com/tests/ztest/) to compare the two proportions. It says that the number of people who performed the action in B is a statistically significant increase over the number of people who performed the action in A with a p-value of 0.00453.

However I wanted to make sure that:

  • a) I'm using the right test -- I know t-tests are sometimes better tests to use when samples sizes are small
  • b) Statistical significance can even be determined on such a small sample.
  • 1
    $\begingroup$ Previously posted at math.stackexchange.com/q/1403465/18398 $\endgroup$ Aug 20, 2015 at 3:54
  • $\begingroup$ Please do not cross-post. Choose the site where you think your question is most appropriate & delete the other. $\endgroup$ Aug 20, 2015 at 4:06
  • 1
    $\begingroup$ I was told by someone on math.stackexchange.com that I should post here, so I did. Which is the more appropriate site in your eyes? $\endgroup$
    – Josh
    Aug 20, 2015 at 4:13

1 Answer 1


If your sample size is small, you can use exact methods (https://en.wikipedia.org/wiki/Exact_test) See for example https://en.wikipedia.org/wiki/Fisher%27s_exact_test

Note that in the first case, not only that the number of people is small, you have only a single person that performed the action of interest.

If that person hadn't perform that action or just another one would have done it, results would have been very different with naive measures.

You might consider https://en.wikipedia.org/wiki/Additive_smoothing in order to make the results more stable. You can also compute https://en.wikipedia.org/wiki/Confidence_interval on your measure of interest.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.