Updated fixed a typo and gave more clarification on the data itself.
I need to make a change to a running online system that i have.
So, I need a method to compare the performance of the system (using a KPI score), before the change (we call this A) and after the change (we call this B).
My plan is as follows:
- split the users into two groups A and B
- keep the original version with group A. Give the changes only to group B.
- take daily measurements for group A and for group B
- compare which group shows better measurements
So far I will have this data:
My raw data collected consists of the number of navigations and the number of clicks per user per day. I perform additional calculations to get a simpler table of [day, ExperimentA, ExperimentB]. Both expA and expB are in the range [0,1] (they are a KPI or a score that i calculate as measurement of performance for my online system for that given date).
Here is an example of generated table (one row per day):
Date, Experiment A, Experiment B
2017/10/01, 0.8, 0.5
2017/10/02, 0.65, 0.7
2017/10/03, 0.7, 0.7
- How can I check if B is better than A? I guess the answer would be to run a t-test?
- If yes, what should use? an independent t-test or related t-test?
thanks a lot