How to embed measurement uncertainty in a two-proportion z-test?

Question

Online marketers often need to choose a "winning" variation among, say, 2 possible variations.

Example:

• PageVariation1 had 1000 sessions with 20 conversions (2.0% response rate) and
• PageVariation2 had 800 sessions with 8 conversions (1.0% response rate).

I'm using the "unpooled" Two-Proportion z-Test for estimating the P-value. In the example above it is 0.9615 - so I might conclude that there's a 96% that PageVariation1 has a higher response rate. Let's say it's good enough to declare it as "winner".

Problem is that some sessions and conversions weren't tracked to the PageVariation level - i.e. I know there were, say, additional 200 sessions and 3 conversions, but those can't be allocated to any of the above. Intuitively it means that there is higher uncertainty due to tracking/measurement issues. Consequently the P-value should be lower.

Question:

• Which test should be used (and how) in order to calculate the new P-value?

Answer 1

It can help to survey the scene: that is, to understand the consequences for your decision of the possible states of affairs.

The missing information can be described by two integers: $m$, the number of additional sessions for "PageVariation2", and $k$, the number of additional conversions for PageVariation2. For each possible combination of $m$ and $k$ ($k = 0, 1, 2, 3$ and $0 \le m \le 200-k$) we can compute what the p-value would be.

In this plot the graph of the p-value versus $m$ for $k=3$ is shown in blue (the highest one), for $k=2$ in red, for $k=1$ in orange, and for $k=0$ in green. For reference, a (two-sided) p-value of 0.05 is shown with a black line.

It is evident that the p-values can range tremendously from indicating strong significance (for $m=197$ and $k=0$) to indicating no significance at all (for small $m$ and $k = 1, 2, 3$). This tells us that your assumptions about the missingness completely determine the significance or lack of significance of the results.

If you think it highly likely that $m$ is large--so that the majority of missing results are for PageVariation2--and that $k$ is small--so that the majority of missing conversions are for PageVariation1--then you can conclude the differences in conversion rates are not likely due to chance and proceed accordingly. If you do not have sufficient information to justify these assumptions, then the data do not provide strong evidence for a significant difference. That is, what you have observed could be the result of random variation rather than due to some intrinsic difference between the two variations.

Answer 2

A few quick points:

• You could draw on the general literature on missing data.
• If the 200 sessions with missing data are known to be completely random, then you could choose to ignore these observations and just analyse the data where you have complete data.
• In general it is important to think through what mechanisms might explain the missing data.
• You may find it useful to read up on strategies for dealing with missing data.