I work for a fundraising organization and we want to implement an A/B test to determine if a particular donor recovery technique is working. The population under study is our lapsed donors. These are donors that gave to us in the past but did not give during our last campaign.
For the study we plan to solicit all of our lapsed donors. All lapsed donors will get the same solicitation package but the A group (test group) will also get some sort of handwritten personalized note. The B group will get no personalized note. Success of the technique will be determined by comparing the number (proportion) of 'recovered' donors in each group. My challenge is to determine how many personalized notes we have to write. For argument sake, I'll say we have 8000 lapsed donors so it is not realistic to simply divide the group in two and write 4000 personalized notes (we don't have the resources).
What tests do I need to run to determine the size of the test group? I would have assumed power analysis but does it matter that I am not doing inferential stats here? We will be studying all lapsed donors. I've done some reading and I understand that there is a lot of nuance here. For example, one could argue that, in a way, I will be doing an inferential test because I want to know that the test will inform our approach to, not only the population under study, but all future lapsed donors as well (the hypothetical super-population?). But even if we set the inferential bit aside, I would still want a robust enough number in the test group to be confident the post test results are reliable.
Please advise or suggest resources. Also note that I am fairly proficient in R so feel free to add R code. I've also explored the 'pwr' package a bit.
Many thanks in advance.