3
$\begingroup$

I'm very new to the idea of A/B testing and I want to see if my train of thought here makes sense.

Suppose that I run an experiment with two designs. I get two sets of resulting data, one for each design. My resulting data has a number of variables that indicate user behavior/interaction with the design, such as how many times a user used the product, ratings given by and of the user, etc. In order to test if the two resulting data set is statistically significantly different, I run a logistic regression with lasso regularization (to deal with the problem of multicollinearity between my variables) to predict which design was seen by each user. I'm making a couple decisions here:

1) Instead of defining one metric (a "conversion" rate), and running a proportions test or a t-test on it, I want to use the coefficients on the logistic regression to see if there are interesting differences between the two designs on each of my "features" (indicating an aspect of user interaction).

2) I'm assuming that the lasso is a good way to deal with correlated features in logistic regression, and that the resulting coefficient can still be interpreted per point 1).

3) If a logistic regression model can predict which design was seen by each user better than chance (+50% accuracy), I can conclude that the two designs have significantly different impact.

Are these three lines of reasoning correct? If not, what's a better way to approach this issue of not just testing one metric but looking at the resulting set of indicators for variations in user behavior? Also, is this the normal use of logistic regression in this setting or is it generally used for a different purpose?

Thanks in advance!

$\endgroup$
2
$\begingroup$

In A/B testing one user is generally shown only one design & you check the conversion against control to see if A) There was a lift if yes then B) which design is better.

If your variables are about the users (like age gender) & design aspects (like color & size etc) then by combining data for both designs with conversion (could be time spent or metric of your choice) you can see which aspects of the design are user dependent.

e.g., - The design A is found more successful because of majority of teenagers visit the website but all people above 35 have liked design B, this interaction will not be captured by model if you take conversion as dependent variable. But doing two individual models will be able to identity this.

It will be good, if you can tell us more about the variables.

Hope this helps.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ thanks for the answer! my variables are not about user demographics or design but about user behavior - such as how long they use the product for each time, how frequently they use it, what ratings they give or receive, what time of week they use the product, etc. What do you mean by doing two individual models? Do you mean that a logistic regression allows me to segment the data and tease out the effects of each feature? Does it apply to features on user behavior? $\endgroup$ – xyy Jul 19 '16 at 14:36
  • $\begingroup$ Instead of combining the data for two design use tow separate models to find out important features. Yes the user behavior parameters will work, no issues there. One Note though - IMO unless you are not able to get the insights using EDA (like univariates & bivariates) only then try using model to find important features. $\endgroup$ – wololo Jul 19 '16 at 17:26
0
$\begingroup$

Doesn't A/B testing imply to have a single variable indicating the treatment? There is no need for regression, particularly not any regularization. You seem to want to compare the group distributions of the other variables to see if your randomization (which you hopefully did) worked out? There are statistical tests for it, e.g. the Kolmogorov-Smirnov test.

| cite | improve this answer | |
$\endgroup$
  • 4
    $\begingroup$ Regression can increase your power. $\endgroup$ – Jeremy Miles Dec 16 '16 at 6:30
0
$\begingroup$

When doing an A/B test you are not restricted to only look at single metric, say, conversion rate. You can compare other metrics capturing user behavior as well. Just run a t-test to see if the metric differs across groups.

This however only applies to user behavior metrics for the time after your test. If you want to judge whether pre-test behavior is a good predictor for an uplift in your key metric, perform a subgroup analysis, e.g. if you want to know if the uplift is only due to the Most engaged customers reacting positively, compare segments in both design groups with each other.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.