3
$\begingroup$

and apologies if this is an overly simple question. I am doing an A/B mailing (where we send out two different solicitation letters). It's going out to past donors so we have some information on them and I have a master list of about 18,000 eligible donors. The point is to see which letter produces more and/or greater donations. Should I generate the two groups completely randomly, or should I sort them by amount given in the past year and manually assign them to two groups taking care that each group had the same distribution of people with various donation histories (same number of 'high donors', 'medium donors' and 'low donors'). What is the best way to create two equivalent groups so that examining the amounts donated later will have meaning ?

$\endgroup$
  • $\begingroup$ To be clear, I am sending either letter out to all 18,000, it's just a matter of how to assign the letters that's the issue. My chief concern is that one group doesn't get many more of the rarer big donors than the other group. $\endgroup$ – chrisfs Aug 13 '13 at 18:58
3
$\begingroup$

Let me begin with the caveat that I'm not a sampling person. I know sampling people, and they're all much smarter than I am, which says something about how simple I am or how non-"overly simple" sampling questions tend to be.

You can do something in-between manually enforcing near-equal distributions of donor amounts and simple random sampling (SRS): stratified random sampling entails separating your donors into groups (so maybe small, medium, and large dollar) and then randomly assigning the donors in those groups to your treatments. Details can be found in Thompson (2012).

The benefit of stratification is that you usually get smaller estimator variances compared to SRS. This is especially true when you have highly different experimental units (e.g. a few high value donors but many low value donors). The down side is that the analysis is a little bit more complex.

A further benefit (that you don't get from poststratification) is that you'll have balanced samples to be able to do inference within each strata (i.e. did treatment A work better for high-value donors but treatment B better for low-value donors) and if the relative cost of using multiple mailers is low these are questions you'll likely want to ask. Again, I'd refer you to Thompson for details.

Thompson, S. (2012). Sampling. 3ed. Wiley.

$\endgroup$
  • $\begingroup$ Went with stratified approach, boss liked the idea. $\endgroup$ – chrisfs Aug 14 '13 at 2:56
1
$\begingroup$

A simple random sample is ideal. You can always segment (or divide) your results based upon their characteristics after the fact.

A great way to divide your results is to just write a script. In python you write something like this:

for contact in contacts:
    if random.randint(0,1) == 0:
        add_contact_info_to_group_a_file(contact)
    else:
        add_contact_info_to_group_b_file(contact)

And just loop through every contact randomly assigning them to group A or B.

Then depending upon the distribution of the amounts you get back you can perform a statistical test for the winner.

$\endgroup$
  • 1
    $\begingroup$ SRS is simpler in construction and analysis, but is it really (in what way is it) ideal? Specifically, how is it better than stratified sampling? $\endgroup$ – neverKnowsBest Aug 13 '13 at 7:06
  • 1
    $\begingroup$ You listed two reasons: simpler in construction and analysis. A third is there's less of a chance of accidentally skewing your experiment. The right tool for the right person and job. In this case, since the OP is apparently new to this, an SRS enables the OP to look at the results both ways after the fact and learn for the future. As I mentioned, you can always stratify (aka segement) after the fact and the OP has three groups that are spread over 9,000 samples in each variation. That means there's probably not a huge chance of random variation that needs to be controlled for. $\endgroup$ – Justin Bozonier Aug 13 '13 at 11:30
  • $\begingroup$ Can you expand on what you mean by stratifying afterwards ? As neverknowsbest mentions, we have a small number of big donors and a large number of small donors. If one group gets a larger number of the rarer big donors, it's going to sway the results simply because big donors give more and sometimes more regularly, that would be an artifact not related to the letters themselves and one I want to avoid. $\endgroup$ – chrisfs Aug 13 '13 at 18:19
  • 1
    $\begingroup$ What is a "small number"? My context is I run split tests on thousands of subjects at a time so I'm used to being able to stratify my results afterward. What I mean by that is after you collect your data then split up your results by donor size. $\endgroup$ – Justin Bozonier Aug 13 '13 at 22:04
  • 1
    $\begingroup$ Out of a master pool of 18,000, I have 400 that are between $600 to $6000 in donation for last year,so it's pretty skewed. I want equal numbers in both groups. $\endgroup$ – chrisfs Aug 14 '13 at 3:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.