When I have strong evidence about the effect of an experiment group in the upper funnel, but less clear evidence at the end of the funnel, how do you go about incorporating the observations from the upper funnel changes into the end funnel calculation?
To further explain what I mean by "funnel", I'm referring to a marketing or sales funnel, like the sequential steps on the path to purchase (or other goal/conversion event). So in the context of an e-commerce site, Landing Page(P1)->Add to Cart(P2)->Checkout Page(P3)->Payment Info(P4)->Checkout completed(P5) where each step is one of the arrows between pages. "Upper funnel" would refer to steps earlier in the sequence, and "bottom funnel" or "end of the funnel" would refer to the final step.
Example situation:
- The user journey is a funnel based on three pages, so P1->P2->P3. P1->P2 is Goal 1 (G1), and P2->P3 is Goal 2 (G2).
- G1 typically occurs for about 40% of visitors, G2 occurs for about 1% of visitors (total visitors).
- G2 is the thing I want to happen most (a checkout), and can only happen after G1.
- I run an experiment with a change, A(control) and B(experiment).
- The experiment is a change to P1, but the hypothesis is that it will improve G2, while G1 stays the same. Since it's a change in P1, it's nevertheless possible G1 will be negatively impacted (it's also possible that G1 will improve, but let's assume that is very unlikely).
- Because the typical conversion rate is so much higher for G1, I know that it will require much less traffic to detect any effect for G1 than it would for G2 (assuming the effect size was equal).
Experiment results look like this:
- A (control)
- Visitors: 35000
- G1 Completions: 14000
- G2 Completions: 350
- B (experiment)
- Visitors: 35000
- G1 Completions: 13790
- G2 Completions: 357
If I look at G1 in isolation (Visitors & G1) with Beta(1,1) as my prior, it is ~95% chance that A beats B.
If I look at G2 in isolation (Visitors & G2) with Beta(1,1) as my prior, it is ~60% chance that B beats A.
Knowing that G2 can't happen without a user first completing G1, and knowing what I know about the observation of the change in G1, the isolated analysis for G2 feels misguided. However, I have no idea how to incorporate this information together in a single inference.
My only guess was to calculate G2, but use different priors for A and B informed by their performance in G1 (rather than Beta(1,1)). But even if this is right, by what process do I establish what those priors are?
I know that a reasonable answer would be "just run the experiment for longer until the answer is clear", but lets assume that traffic is low enough that it's worthwhile to make some kind of assessment at this point as to the potential -- especially because the low base rate for G2 means that it will take a lot more traffic to be certain about the impact, and you'd be waiting all that time knowing that G1 will be performing worse.
