I am working on evaluating an experiment or A/B test but I am fairly new to it. In that process, I am trying to interpret which post-test metrics are significant and the methodology to decide that. The experiment tested the impact of an in-app notification on several user-centric metrics. Both, the treatment and control group consisted of 50,000 users.
Here is a table with the current output:
Metric | P-value | Effect
Time Spend/User | 0.31 | 0.17%
Revenue/User | 0.01 | 0.02%
Session/User | 0.04 | 2.3%
Uninstalls | 0.02 | -0.3%
My first reaction was to check if p-value<=0.05 (significance) and if so, then the null hypothesis shall be rejected. However, I am unsure how to bring in the Effect aspect as some look relatively small and may not cover the cost of implementing and maintaining this new feature.
Questions
- Is there any systematic methodology on how to assess the practical significant of each metric based on both, p-value and Impact instead of p_value only?
- What holistic decision shall be made based and what is the justification to do so?
I hope I can get some guidance. I truly appreciate it.