A/B test with two metrics, we test one metrics for improvement on the condition the other one doesn't change

Question

I have the following business request:

"Test this change on the landing page and implement it for all the visitors if the test shows improvement in the conversion rate, but if the page load speed significantly differs between the two variations, discard the experiment results and investigate"

The rationale behind it is we want to test the change, but we also want to make sure the results we are getting can only be attributed to the visual change itself, and not the change in page load speed that may occur due to poor implementation of A/B testing framework etc or some other factors.

As far as I know in our case page load speed is what they call "sanity guardrail metric".

However, I am not sure how to translate it into a proper A/B test.

What I figured out so far:

1. We choose one-tail test for testing the conversion rate (right tail in our case, since we test for an improvement)
2. We chose two-tailed test for the page load metric, since we test for a change in any direction.

What I have my doubts about:

1. Required sample size: since we are going to monitor 2 separate metrics, one of them being a continuous metric and the other one - proportion, I am not sure how to calculate the required sample size. Intuition tells me that I should calculate the required sample size for both metrics with the specified minimum detectable effect, power and error probability and choose the larger sample size. Of course I expect the sample size for page load metric to be larger.
2. Should I use some sort of correction here? Since I am going to monitor two metrics, maybe I need to apply some sort of correction (bernoully, dunett etc.), but I have doubts, since we do not seek to improve the page load speed - we only need to make sure it doesn't change, but since the page load speed metric is a subject to false positive as well, I guess something must be done about it as well. Besides, we combine one-tail metrics and two-tailed one into one test, which further complicated things.

Any help will be much appreciated, as well as any links to the relevant resources that cover similar problems.

Answer 1

1. Unless it's particularly difficult to test a larger sample, you should generally opt for the sample size required for the less powerful test. That is, do the power calculations for both tests, and go with whichever sample size is biggest. There's no such thing as "too much power".
2. There's no need for p-value correction here. Correction is necessary where you're looking at multiple outcome measures, and would deem the test to be "successful" if any one of them were to improve, since that would increase the chances of a false positive. I'm defining "false positive" here to be "concluding that the change has increased conversion rates, when it in fact hasn't". The fact that you're also looking at load times doesn't increase the probability of that happening.

Other notes:

• It's almost always possible that your test could reduce conversion rates, so I don't think a one-tailed test on conversions is appropriate.
• You could use a one-tailed test for load times, if you assume there's no way that people could accidentally improve them.