I work for a fundraising organization and we want to implement an A/B test to determine if a particular donor recovery technique is working. The population under study is our lapsed donors. These are donors that gave to us in the past but did not give during our last campaign.

For the study we plan to solicit all of our lapsed donors. All lapsed donors will get the same solicitation package but the A group (test group) will also get some sort of handwritten personalized note. The B group will get no personalized note. Success of the technique will be determined by comparing the number (proportion) of 'recovered' donors in each group. My challenge is to determine how many personalized notes we have to write. For argument sake, I'll say we have 8000 lapsed donors so it is not realistic to simply divide the group in two and write 4000 personalized notes (we don't have the resources).

What tests do I need to run to determine the size of the test group? I would have assumed power analysis but does it matter that I am not doing inferential stats here? We will be studying all lapsed donors. I've done some reading and I understand that there is a lot of nuance here. For example, one could argue that, in a way, I will be doing an inferential test because I want to know that the test will inform our approach to, not only the population under study, but all future lapsed donors as well (the hypothetical super-population?). But even if we set the inferential bit aside, I would still want a robust enough number in the test group to be confident the post test results are reliable.

Please advise or suggest resources. Also note that I am fairly proficient in R so feel free to add R code. I've also explored the 'pwr' package a bit.

Many thanks in advance.

  • $\begingroup$ Are you interested in mean $ given or proportion of donors who resume giving? What size difference between means or proportions would be meaningful for you? Amongst givers, what is the s.d. of the contribution, or what is the proportion that typically responds to an appeal? $\endgroup$ – Joel W. May 23 '14 at 19:54
  • $\begingroup$ I'm interested in the proportion of donors who resume giving (2nd paragraph above). I may look at other factors as well, but let's keep it simple for now. As for the size difference, let's say a 5% increase in the test group would be meaningful. About 15% of our donors respond to appeals. Sounds like you are going the power analysis route? But please also comment on the issues I raise in paragraph 3 (about inferential/non-inferential stats) if you have any expertise. Thanks. $\endgroup$ – jtdoud May 23 '14 at 20:20
  • 1
    $\begingroup$ If you're studying all lapsed donors, power analysis is a bit irrelevant (I'd have though), because you're going to do them all, and you can't get a bigger sample, even if you want to. I also agree that the population of interest is all lapsed donors in the future, and so even though you are, in a sense, including the whole population that is available to you, you are not using the whole population and therefore you should do inferential statistics. $\endgroup$ – Jeremy Miles May 24 '14 at 0:42
  • $\begingroup$ But even if we consider my group of lapsed donors as the 'whole' population... In the case above, I would not want to use only 10 people in the test group for obvious reasons. I guess this is getting closer to the crux of my question. Is power analysis the right tool in this situation for determining the minimum size of my test group? $\endgroup$ – jtdoud May 24 '14 at 19:48
  • $\begingroup$ Perhaps you would be most happy with a large enough sample to create a small confidence interval. Personnally, I would use power analysis. $\endgroup$ – Joel W. May 25 '14 at 13:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.