Updated fixed a typo and gave more clarification on the data itself.

I need to make a change to a running online system that i have.

So, I need a method to compare the performance of the system (using a KPI score), before the change (we call this A) and after the change (we call this B).

My plan is as follows:

  • split the users into two groups A and B
  • keep the original version with group A. Give the changes only to group B.
  • take daily measurements for group A and for group B
  • compare which group shows better measurements

So far I will have this data:

My raw data collected consists of the number of navigations and the number of clicks per user per day. I perform additional calculations to get a simpler table of [day, ExperimentA, ExperimentB]. Both expA and expB are in the range [0,1] (they are a KPI or a score that i calculate as measurement of performance for my online system for that given date).

Here is an example of generated table (one row per day):

Date, Experiment A, Experiment B

2017/10/01, 0.8, 0.5

2017/10/02, 0.65, 0.7

2017/10/03, 0.7, 0.7


  1. How can I check if B is better than A? I guess the answer would be to run a t-test?
  2. If yes, what should use? an independent t-test or related t-test?

thanks a lot

  • $\begingroup$ Describe your data - is it percentage of users - complaints or index for efficiecy of different machines /users . total sample and subsamples- number for each group ? $\endgroup$ – Subhash C. Davar Nov 15 '17 at 16:29
  • $\begingroup$ you can think of the raw data collected as a number of navigations and clicks per user per day. I already do additional calculations and get a simple table of <day, expA, expB>. It doesn't matter much what are the valid values for expA and expB but they both have same type and range of values $\endgroup$ – Samer Aamar Nov 16 '17 at 6:00
  • $\begingroup$ @subhashc.davar i have updated the question and reflected your comment. I had a mistake in the example i brought to you. fixed now. $\endgroup$ – Samer Aamar Nov 16 '17 at 6:07
  • $\begingroup$ thanks! But, I can't understand what is better measurements ? $\endgroup$ – Subhash C. Davar Nov 16 '17 at 6:23
  • $\begingroup$ @subhashc.davar The higher the values are, the better the system is. Assume, for simplicity, we get in ExpB a fixed score of 0.9 every day but for ExpA we get a fixed score of 0.5 every day, this means that system B is better than A. In this simple example I would say that system B is improving our system by 0.4 score value, which is 40%. $\endgroup$ – Samer Aamar Nov 16 '17 at 7:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.