1
$\begingroup$

Suppose I run an experiment which tests which version of an email is most likely to cause people to click on a survey link inside the email.

There are two experimental manipulations: the email subject and the text within the email. For each experimental manipulation there are two levels (treatment and control). Both experimental manipulations are randomized independently of each other.

Normally (assuming no interaction effects), one would simply estimate treatment effects by running the following regression:

$ClickedLink_i = \beta_0 + \beta_1 EmailSubject_i + \beta_2 EmailText$

Where ClickedLink is a binary variable equal to 1 if user i clicked on the survey link, 0 otherwise. EmailSubject and EmailText are also binary variables, equal to 1 if the user received the Treatment email subject and email text respectively. (It may actually be more appropriate to use logit/probit models, and feel free to use those when answering, but this isn't the main point of the question)

I notice that email subject actually affects whether users actually see the email text. That is, if users don't click on the email after reading the subject line, the text of the email won't matter. How can I reformulate my regression to take this into account?

$\endgroup$
  • $\begingroup$ Do you know whether a user saw the email text or not? $\endgroup$ – mkt - Reinstate Monica Oct 9 '19 at 12:04
  • $\begingroup$ Not totally sure, but this thread might be relevant: stats.stackexchange.com/q/372257/121522 $\endgroup$ – mkt - Reinstate Monica Oct 9 '19 at 12:06
  • $\begingroup$ @mkt Good question. Let's assume we can't (the analysis seems pretty straightforward if we can) $\endgroup$ – wwl Oct 9 '19 at 12:13
4
$\begingroup$

This is really quite straight-forward --- you just include the interaction term in your regression model. By having a control and experimental group for two separate parts of the message, you effectively have four possible messages, and your experiment is seeking to find the effect of each of these four messages. So just treat this as an RCT with four experimental groups, and make an inference for all those groups (by including an interaction term in your model). If it is indeed the case that the subject message affects whether the user reads the text, then that should show up in your data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.