I have the following business request:

"Test this change on the landing page and implement it for all the visitors if the test shows improvement in the conversion rate, but if the page load speed significantly differs between the two variations, discard the experiment results and investigate"

The rationale behind it is we want to test the change, but we also want to make sure the results we are getting can only be attributed to the visual change itself, and not the change in page load speed that may occur due to poor implementation of A/B testing framework etc or some other factors.

As far as I know in our case page load speed is what they call "sanity guardrail metric".

However, I am not sure how to translate it into a proper A/B test.

What I figured out so far:

  1. We choose one-tail test for testing the conversion rate (right tail in our case, since we test for an improvement)
  2. We chose two-tailed test for the page load metric, since we test for a change in any direction.

What I have my doubts about:

  1. Required sample size: since we are going to monitor 2 separate metrics, one of them being a continuous metric and the other one - proportion, I am not sure how to calculate the required sample size. Intuition tells me that I should calculate the required sample size for both metrics with the specified minimum detectable effect, power and error probability and choose the larger sample size. Of course I expect the sample size for page load metric to be larger.
  2. Should I use some sort of correction here? Since I am going to monitor two metrics, maybe I need to apply some sort of correction (bernoully, dunett etc.), but I have doubts, since we do not seek to improve the page load speed - we only need to make sure it doesn't change, but since the page load speed metric is a subject to false positive as well, I guess something must be done about it as well. Besides, we combine one-tail metrics and two-tailed one into one test, which further complicated things.

Any help will be much appreciated, as well as any links to the relevant resources that cover similar problems.

  • $\begingroup$ Could you not just do a single regression with change + loading speed as independent variables and you are interested in change coefficient $\endgroup$
    – seanv507
    Commented Mar 21, 2023 at 13:58

1 Answer 1

  1. Unless it's particularly difficult to test a larger sample, you should generally opt for the sample size required for the less powerful test. That is, do the power calculations for both tests, and go with whichever sample size is biggest. There's no such thing as "too much power".
  2. There's no need for p-value correction here. Correction is necessary where you're looking at multiple outcome measures, and would deem the test to be "successful" if any one of them were to improve, since that would increase the chances of a false positive. I'm defining "false positive" here to be "concluding that the change has increased conversion rates, when it in fact hasn't". The fact that you're also looking at load times doesn't increase the probability of that happening.

Other notes:

  • It's almost always possible that your test could reduce conversion rates, so I don't think a one-tailed test on conversions is appropriate.
  • You could use a one-tailed test for load times, if you assume there's no way that people could accidentally improve them.
  • $\begingroup$ I like this answer, but disagree with the last two suggestions. The speed hypothesis test is the canary in the experimental mine. If it rejects the null of no change, then the conversion test does not matter because it would be biased. The number of conversion tails does not matter in that scenario. $\endgroup$
    – dimitriy
    Commented Jun 28 at 0:35
  • $\begingroup$ Let's say we cannot reject the null that the site speed changed. The canary lives, so we proceed to look at conversion. What matters is that we reject the null that the conversion effect is negative or zero in favor of the alternative that it is positive. In this scenario, a two-sided test seems wasteful in terms of the time it takes to get to a ship decision, users exposed to a bad experience, and made unavailable for other experiments. $\endgroup$
    – dimitriy
    Commented Jun 28 at 0:36
  • $\begingroup$ Having highly precise knowledge of the extent to which an effect is harmful does not change the ship decision. The extra sample and time needed for a two-sided test would be wasted. It would be better to spend that alpha to improve power on the right. $\endgroup$
    – dimitriy
    Commented Jun 28 at 0:37
  • $\begingroup$ Moreover, if the visual change is subtractive, site speed could improve. Suppose we cannot reject the null that the speed effect is positive. We go on to do the conversion test and find that a significant lift. But then the conversion effect could be due purely to the side speed improvement and not visual stuff. It seems safer to stick with the two-sided test if a clear picture is needed. $\endgroup$
    – dimitriy
    Commented Jun 28 at 0:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.