I am prepping for an interview and here is one of the questions I have come across:

How would you conduct an A/B test on an opt-in feature?

I think you cannot perform a "fair" test with an opt-in option. People viewing the A version of the site will come from the whole distribution of users whereas people viewing B will only come from the set who have opted in. This seems to violate the conditions for a controlled experiment.

What is your take on this?

  • $\begingroup$ What is opt-in ? As far as I know, opt-in means that the user has actively agreed e.g. on using his email address for newsletter purpose. Given that, does the question refer to a test for a feature only opt-in users can see ? $\endgroup$ – steffen Apr 29 '14 at 11:11
  • $\begingroup$ @steffen - Yes, I assume the question means you only see the test feature if you have previously opted-in. $\endgroup$ – mchangun Apr 30 '14 at 3:26

Regarding your statement

You are correct: If the test shall compare an opt-in version of a site to a non-opt-in-version, then one cannot perform a fair test. Since visitors how have opted-in are probably more engaged than regular users, I'd expect the group of opt-in visitors to outperform the other group in engagement related metrics as long as meaningful versions are compared (although such a test may uncover that the opt-in version is not good enough). It might even be that the metrics for different versions are not comparable at all (because of different goals).

Another interpretation of the interview question

Since you have used the self-study-tag, I'd provide some starting points for further thinking instead of a complete answer (I can answer them myself if requested though, otherwise I suggest that you provide an answer to your own question).

Here are two more interpretations of the interview question

  1. How to conduct an A/B-Test for a feature that shall lead to an opt-in (i.e. the visitors have not opted in yet but the feature shall make them do so) ? This would be a more standard A/B-Test setup so I guess it is unlikely the question was meant this way.
  2. How to conduct an A/B-Test restricted to test different versions of an opt-in feature. This would be my first guess how the question is meant (more complicated). So let's focus on this one.

So how to conduct an A/B-Test for the second case ? Consider that the site with the opt-in feature can be in an restricted area (i.e. you have to e.g. log in to get there) or can be just a different version of a site every visitor can access (e.g. the homepage).

  • $\begingroup$ Thanks for the pointers. Upon re-reading of the question, I think my original interpretation was wrong and your interpretations seems much more like it. $\endgroup$ – mchangun May 1 '14 at 8:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.