Can I use t-test to compare two groups whose event is rare (less than 1 per thousand)?

Question

Background

We ran a test marketing campaign at my company to see if our new customer segmentation has an impact on the response rate to our offer.

We have created two groups to run this test marketing campaign:

• Group 1: 2,000 people randomly selected among existing clients.
• Group 2 : 10,000 people that were selected following the new segmentation process.

For each of this group, we obtained the following number of people who responded to the campaign:

• Group 1: 3 people out of 2,000
• Group 2: 5 people out of 10,000

So we are dealing with a really rare case type of problem. My goal is to understand it there is a significant difference between the response rate of group 1 and group 2.

Questions

1) Can I run a t-test with such a rare case type of problem?

I suspect that this is not possible because the data is highly skewed (see this paper: http://advan.physiology.org/content/34/3/128)

2) Which test (possible to be ran with R) should I use to compare the mean of my two groups?

3) Is there a rule of thumb to determine if the t-test is not appropriate?

Any help appreciated!

Answer 1

Fisher's exact test on this table:

                     Group
                --------------
                |   I  |  II  |
          ------|------|------|
Response  | Yes |    3 |    5 |
          |  No | 1997 | 9995 |
          ---------------------

The t-test will fail very far from 50-50. The chi-square test will fail with sparse counts. Fisher's will work (in R, this is fisher.test).