1
$\begingroup$

I run a campaign with an experimental design setup. The target and control group is representative and split 70/30.

The target group receives an email communication

The dynamics of the campaign require the user to register for the offer via the email and then make a purchase

(The control group has no registration process)

Therefore, the possible scenarios are as follows

Target | registered | converted
Target | registered | ! converted

Target | ! registered | converted
Target | ! registered | ! converted

Control | converted
Control | ! converted

Note - Target | ! registered | converted are users who may or may not have seen the email but arrive at the website and purchase anyway. Only those who register actually avail the offer.

Question

In order to measure the impact of the email on conversion rate & total spend, which group/subgroup would be compared with the control group

$\endgroup$
0
$\begingroup$

I understand the questions to be:

  1. Does the additional step of registration complicate the treatment effect analysis?

You can compare the conversion rate/spend between the control and treatment group directly. Your estimate will then be conditional on the registration process. In other words, you predict the increase in conversion probability given an email with an offer for which the user has to register.

However, if you make a cost-based decision to target, then you will have to include in your decision the chance that the user actually registers and uses the offer. I'd suggest you subtract the actual offer cost from the net return of each customer.

  1. Can I estimate the treatment effect on the conversion rate/spend if some customer overlook the treatment email?

If customer assigned to the target group might not receive or interact with the treatment, your estimate will be the intention to treat effect. This is a common problem in medicine, where not all patients take their prescribed medicine correctly.

Your estimate will be lower than the pure treatment effect, since customers that don't receive the email will be counted as treated but will not show any effect. Your model will include the chance that a customer sees the email, so targeting will favor customer more likely to look at and act because of the email, which is likely what you want from targeting.

(If you want to compare different email headers or improve opening rates in others ways, you might want to come back and try to disentangle the effect in the future.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.