# Completely random or structured selection of receipients for an A/B mailing

and apologies if this is an overly simple question. I am doing an A/B mailing (where we send out two different solicitation letters). It's going out to past donors so we have some information on them and I have a master list of about 18,000 eligible donors. The point is to see which letter produces more and/or greater donations. Should I generate the two groups completely randomly, or should I sort them by amount given in the past year and manually assign them to two groups taking care that each group had the same distribution of people with various donation histories (same number of 'high donors', 'medium donors' and 'low donors'). What is the best way to create two equivalent groups so that examining the amounts donated later will have meaning ?

• To be clear, I am sending either letter out to all 18,000, it's just a matter of how to assign the letters that's the issue. My chief concern is that one group doesn't get many more of the rarer big donors than the other group. – chrisfs Aug 13 '13 at 18:58

Let me begin with the caveat that I'm not a sampling person. I know sampling people, and they're all much smarter than I am, which says something about how simple I am or how non-"overly simple" sampling questions tend to be.

You can do something in-between manually enforcing near-equal distributions of donor amounts and simple random sampling (SRS): stratified random sampling entails separating your donors into groups (so maybe small, medium, and large dollar) and then randomly assigning the donors in those groups to your treatments. Details can be found in Thompson (2012).

The benefit of stratification is that you usually get smaller estimator variances compared to SRS. This is especially true when you have highly different experimental units (e.g. a few high value donors but many low value donors). The down side is that the analysis is a little bit more complex.

A further benefit (that you don't get from poststratification) is that you'll have balanced samples to be able to do inference within each strata (i.e. did treatment A work better for high-value donors but treatment B better for low-value donors) and if the relative cost of using multiple mailers is low these are questions you'll likely want to ask. Again, I'd refer you to Thompson for details.

Thompson, S. (2012). Sampling. 3ed. Wiley.

• Went with stratified approach, boss liked the idea. – chrisfs Aug 14 '13 at 2:56

A simple random sample is ideal. You can always segment (or divide) your results based upon their characteristics after the fact.

A great way to divide your results is to just write a script. In python you write something like this:

for contact in contacts:
if random.randint(0,1) == 0:

• Out of a master pool of 18,000, I have 400 that are between $600 to$6000 in donation for last year,so it's pretty skewed. I want equal numbers in both groups. – chrisfs Aug 14 '13 at 3:38