and apologies if this is an overly simple question. I am doing an A/B mailing (where we send out two different solicitation letters). It's going out to past donors so we have some information on them and I have a master list of about 18,000 eligible donors. The point is to see which letter produces more and/or greater donations. Should I generate the two groups completely randomly, or should I sort them by amount given in the past year and manually assign them to two groups taking care that each group had the same distribution of people with various donation histories (same number of 'high donors', 'medium donors' and 'low donors'). What is the best way to create two equivalent groups so that examining the amounts donated later will have meaning ?
Let me begin with the caveat that I'm not a sampling person. I know sampling people, and they're all much smarter than I am, which says something about how simple I am or how non-"overly simple" sampling questions tend to be.
You can do something in-between manually enforcing near-equal distributions of donor amounts and simple random sampling (SRS): stratified random sampling entails separating your donors into groups (so maybe small, medium, and large dollar) and then randomly assigning the donors in those groups to your treatments. Details can be found in Thompson (2012).
The benefit of stratification is that you usually get smaller estimator variances compared to SRS. This is especially true when you have highly different experimental units (e.g. a few high value donors but many low value donors). The down side is that the analysis is a little bit more complex.
A further benefit (that you don't get from poststratification) is that you'll have balanced samples to be able to do inference within each strata (i.e. did treatment A work better for high-value donors but treatment B better for low-value donors) and if the relative cost of using multiple mailers is low these are questions you'll likely want to ask. Again, I'd refer you to Thompson for details.
Thompson, S. (2012). Sampling. 3ed. Wiley.
A simple random sample is ideal. You can always segment (or divide) your results based upon their characteristics after the fact.
A great way to divide your results is to just write a script. In python you write something like this:
for contact in contacts: if random.randint(0,1) == 0: add_contact_info_to_group_a_file(contact) else: add_contact_info_to_group_b_file(contact)
And just loop through every contact randomly assigning them to group A or B.
Then depending upon the distribution of the amounts you get back you can perform a statistical test for the winner.