Which test should I use for hypothesis testing with a small sample size?

Question

I've run a test with one control and one experiment group, and am questioning myself on whether or not I've used the right test (or if significance can even be calculated on the following sample sizes).

The data is as follows:

The control cohort (A) had 63 people see the treatment and 1 person performed the action (1.59%)

The Experiment cohort (B) had 64 people see the treatment and 9 people performed the action (14.1%)

I used a z-test for two population proportions (this equation: http://www.socscistatistics.com/tests/ztest/) to compare the two proportions. It says that the number of people who performed the action in B is a statistically significant increase over the number of people who performed the action in A with a p-value of 0.00453.

However I wanted to make sure that:

• a) I'm using the right test -- I know t-tests are sometimes better tests to use when samples sizes are small
• b) Statistical significance can even be determined on such a small sample.

Answer 1

If your sample size is small, you can use exact methods (https://en.wikipedia.org/wiki/Exact_test) See for example https://en.wikipedia.org/wiki/Fisher%27s_exact_test

Note that in the first case, not only that the number of people is small, you have only a single person that performed the action of interest.

If that person hadn't perform that action or just another one would have done it, results would have been very different with naive measures.

You might consider https://en.wikipedia.org/wiki/Additive_smoothing in order to make the results more stable. You can also compute https://en.wikipedia.org/wiki/Confidence_interval on your measure of interest.