How to best evaluate A/B Test result for statistical and practical significance holistically?

Question

I am working on evaluating an experiment or A/B test but I am fairly new to it. In that process, I am trying to interpret which post-test metrics are significant and the methodology to decide that. The experiment tested the impact of an in-app notification on several user-centric metrics. Both, the treatment and control group consisted of 50,000 users.

Here is a table with the current output:

Metric          | P-value | Effect
Time Spend/User | 0.31    | 0.17%
Revenue/User    | 0.01    | 0.02%
Session/User    | 0.04    | 2.3%
Uninstalls      | 0.02    | -0.3%

My first reaction was to check if p-value<=0.05 (significance) and if so, then the null hypothesis shall be rejected. However, I am unsure how to bring in the Effect aspect as some look relatively small and may not cover the cost of implementing and maintaining this new feature.

Questions

1. Is there any systematic methodology on how to assess the practical significant of each metric based on both, p-value and Impact instead of p_value only?
2. What holistic decision shall be made based and what is the justification to do so?

I hope I can get some guidance. I truly appreciate it.

Answer 1

A lot information are missing to give a definite answer. I'll address some points coming to my mind

Effect size

Effect size is not unambiguously defined (https://en.wikipedia.org/wiki/Effect_size). In your case, it looks like the expected lift from the new variation (i.e. $\frac{mean_{variation} - mean_{control}}{mean_{control}}$)

If this is true, I would highly doubt the p-values (e.g. revenue). I played around assuming a binomial distribution (which has less variance than revenue and hence less uncertainty), but I was not able to reproduce it (checkout e.g. Evan Miller's Tool, from abtestguide or better, a verified software library).

Maybe the effect size is calculated in another way ? Maybe there is an issue with the execution of the test (e.g. peeking) ?

P-value vs effect size

For now, let's assume the above values are all valid.

In business, the systematic approach boils down to the question, whether the expected profit increase of a new feature is greater than the expected costs.

However ...

• Is Revenue short-term or expected life time value ? Is the A/B-Test duration long enough so this can be measured ? In your case, the uninstall-rate is higher than the revenue increase.
• What is the expected impact on fresh installs and long-term users (which leads to factors like acquisition or retention costs, prospective growth) ?
• What is the affected impact on casual and power-users ? How does this affect the product strategy ?
• Is the goal really profit at this point or getting as much users as fast as possible ? Or maximize time spent in the app ?
• The cost of implementation has already been paid (excluding something like code cleanup). New features mean additional complexity, which may result in increased costs for additional features. This is extremely hard to measure.
• The manager in charge fears its reputation is at stake and needs the new feature to be a success ... which means any contradictory interpretation is irrelevant by definition.