Online marketers often need to choose a "winning" variation among, say, 2 possible variations.
- PageVariation1 had 1000 sessions with 20 conversions (2.0% response rate) and
- PageVariation2 had 800 sessions with 8 conversions (1.0% response rate).
I'm using the "unpooled" Two-Proportion z-Test for estimating the P-value. In the example above it is 0.9615 - so I might conclude that there's a 96% that PageVariation1 has a higher response rate. Let's say it's good enough to declare it as "winner".
Problem is that some sessions and conversions weren't tracked to the PageVariation level - i.e. I know there were, say, additional 200 sessions and 3 conversions, but those can't be allocated to any of the above. Intuitively it means that there is higher uncertainty due to tracking/measurement issues. Consequently the P-value should be lower.
- Which test should be used (and how) in order to calculate the new P-value?