1
$\begingroup$

This is the first time I am designing an A/B experiment and I have read up all I could but still not completely clear on making a decision. It would be really great if the pros here could provide some guidance on the below setup or point me in the right direction

We are building a new capability that will enhance the productivity of employees. Let's say we have 100 sellers. We want to pilot this out in the next quarter to measure the impact using Experiment and Control group (50 sellers each, randomly chosen). I see two options from here on.

  • Run the pilot and compare the productivity metric (let's say average revenue generated per seller) between the Pilot and non-pilot users. (two sample t-test)
  • Run the pilot and compare the productivity metric for the Pilot users in the prior quarter vs quarter of the pilot period. (paired t-test)

These sellers are organized by geographical regions, so a sale for one seller does not effect the sale for other users i.e. they do not directly compete with one another.

$\endgroup$

1 Answer 1

1
$\begingroup$

A randomized experiment into two groups is usually useful at answering a question such as yours.

However, it is useful to compare the same people under two conditions.

I suggest you include both components. This is called a pre - post study with randomized groups. In analysis you can compute the difference between the first period and the second period (the change) for each participant. You then compare these differences between the two different groups. This is likely to use a two-sample t-test or a corresponding nonparametric test.

You might also find analysis of covariance useful, treating the first period measurement as a covariate for the second time period. This often gives more definite conclusions than comparing the changes of the two groups.

$\endgroup$
1
  • $\begingroup$ Thanks for the answer - it makes good sense. By including both the design elements we can note the difference of the differences! (difference between random and control of the differences between individual pre and post measures) $\endgroup$
    – Gvancha
    Jun 9, 2017 at 8:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.