3
$\begingroup$

Suppose I have two versions of an email and each contains a link. I randomly assign two groups to receive the respective emails of sizes $N_1$ and $N_2$ (i.e. group $i$ receives email $i$ for $i=1,2$). Denote the number of group $i$ recipients who open the email as $n_i$ for $i=1,2$. Further, denote the number of group $i$ recipients who clicked the link after opening the emails in the respective groups as $c_i$ for $i=1,2$. I am interested in whether the proportion $\frac{c_i}{n_i}$ is different for the two emails/groups. Two questions:

  1. Can I apply a standard $z$ test given that the samples are not random, i.e. recipients self-select by opening the emails.
  2. If so, my understanding is that I would be using $n_1$ and $n_2$ as sample sizes. Is that correct?

I have tried to search for answers online but am not really sure how to concisely articulate this question. Any journal articles or textbook references would be very helpful. Thanks in advance!

$\endgroup$
4
  • $\begingroup$ The randomization seems to be in the selection of $N_1$ and $N_2.$ The 'payoff' seems to be those those who clicked to see the link, $c_1$ and $c_2.$ I'd consider Treatment $i$ to be the presentation and contents of email $I$ So IMHO the only possibility for a legitimate test is is to compare $p_i = c_i/N_i, i = 1,2.$ $\endgroup$
    – BruceET
    Mar 24, 2021 at 15:16
  • 1
    $\begingroup$ Thanks for your reply! We are considering $n_i/N_i, c_i/N_i,$ and $c_i/n_i$, but $c_i/n_i$ is the test of highest interest. Really looking for a way to test this or find a legitimate source that explains why it's not possible. $\endgroup$
    – dlnB
    Mar 24, 2021 at 15:20
  • 1
    $\begingroup$ If the two emails were exactly the same, then you might argue for $c_i/n_i.$ Certainly, you can run almost any test. The question is whether the results would be meaningful. If you want to explain what's really going on and give the six numbers, maybe further discussion would be warranted. $\endgroup$
    – BruceET
    Mar 24, 2021 at 15:41
  • 2
    $\begingroup$ We are most interested in whether $c_1/n_1=c_2/n_2$. Someone else on my team proposed conducting a standard $z$ test but using $N_1$ and $N_2$ as the sample sizes, but this is clearly wrong. I would like a paper or textbook to demonstrate that logic is wrong (ideally one that also demonstrates the correct approach as well, but at least that this is wrong). $\endgroup$
    – dlnB
    Mar 24, 2021 at 16:04

1 Answer 1

3
$\begingroup$

You might find it useful (and relevant) to model this situation by supposing that among people who open these emails, (1) the decisions to click through are independent and (2) those decisions are made with fixed probabilities $p_i$ which might differ by type of email.

The first assumption is non-controversial (unless you are sending emails in blocks to groups of related people). The second one ought to be tested, but we can at least use it to proceed with some analysis.

I also assume the recipients are blind as to the types of emails: that is, superficially they appear identical, so that the decision to open the email is independent of its type.

The randomized choice of recipients of these emails assures that you have random samples of two (overlapping) populations: namely, those who would open emails of type 1 and those who would open emails of type 2. Assuming these populations are large compared to the numbers of emails sent, the lack of independence in the sampling (nobody has any chance of receiving both types of emails) is inconsequential.

These assumptions place you in the textbook situation of comparing two (hypothetical) probabilities $p_i$ in two populations based on a two-sample Binomial experiment with sample sizes $n_1$ and $n_2,$ thereby confirming both assertions (1) and (2) in the question.

$\endgroup$
6
  • $\begingroup$ If the assumption the recipients are blind as to the types of emails is invalid, (eg. the two emails have different subject lines) , would that effect one's ability to test for a difference in, 'the proportion of email-openers who click on the link'? $\endgroup$
    – Ryan Volpi
    Mar 24, 2021 at 17:12
  • 1
    $\begingroup$ Extremely helpful, @whuber (+1). Thanks! Do you know of any papers I could reference to show my colleagues? While I certainly believe and agree with your answer, a forum answer is not exactly something I can take to my team. $\endgroup$
    – dlnB
    Mar 24, 2021 at 17:15
  • 2
    $\begingroup$ @Ryan Absolutely, because opening the email would be confounded with clicking through. dlnB, if you can't take a forum answer to your team, one wonders why it's even worth using a forum ;-). I constantly hope that a reasoned argument will be more effective and persuasive than any appeal to authority. $\endgroup$
    – whuber
    Mar 24, 2021 at 17:21
  • $\begingroup$ I tested assumption (2) and found that one email has a higher probability of being opened than the other. What is the implication of the result that $p_1 \neq p_2$? $\endgroup$
    – dlnB
    Mar 24, 2021 at 18:16
  • 2
    $\begingroup$ How much greater is the probability and is the difference statistically significant? For this test you are in a classic Binomial proportions setting by design (due to your random splitting of a group of people into two classes). If there is a substantial significant difference then you cannot draw conclusions about the click-through rate per opened email, but you can still draw conclusions about the rate per email sent. $\endgroup$
    – whuber
    Mar 24, 2021 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.