Is peeking in A/B tests bad even if you don't stop your experiment?

Question

I have been asked a test about A/B testing that I'm unsure how to answer.

Say you're running a conversion rate experiment, and my A variation has a conversion rate of 2% and my B variation has a 2.2% conversion rate (so a 10% relative change).

I set up my system to collect data daily, and I calculate the p-value from a chi-squared test daily. However, depite peeking at the p-value daily, I don't stop the experiment until the agreed stopping point (14 days after the experiment started).

My question is: does this affect things like the statistical power/false positive rate?

I simulated the process with the code below:

import pandas as pd
import numpy as np
import dsutil as hfds
from pymc3.stats import hpd
from scipy.special import betaln
from scipy.stats import beta
from scipy.stats import chi2_contingency


class DGP(object):
  
    """
    Data Generating Process
    """

    def __init__(self, arrivals_mean=100, 
                 conversion_rate=0.02, 
                 n_days=14, 
                 n_reps=1,
                 mde=0.0):
        self.arrivals_mean = arrivals_mean
        self.conversion_rate = conversion_rate
        self.n_days = n_days
        self.n_reps = n_reps
        self.mde=mde
    
    def generate_data(self, mde=0.0):
        """
        Simulate a conversion experiment with Poisson arrivals and a conversion_rate conversion probability
        """
        n_arrivals = np.random.poisson(lam=self.arrivals_mean, size=self.n_days*self.n_reps)
        n_converted = np.random.binomial(n_arrivals, self.conversion_rate+(self.conversion_rate*mde))
        iterables = [range(self.n_reps), range(self.n_days)]
        index = pd.MultiIndex.from_product(iterables, names=['replicate', 'day'])
        df = pd.DataFrame(data={'arrivals': n_arrivals, 'converted': n_converted}, index=index)
        return df.cumsum()
    
    def simulate(self):
        control_df = self.generate_data()
        test_df = self.generate_data(self.mde)
        return control_df, test_df


n_trials = 1
days = 13
positives = 0

control_df, test_df = DGP(mde=0.02).simulate()
for day in range(days): 
    n_A = control_df.iloc[day]['arrivals']
    obs_A = control_df.iloc[day]['converted']

    n_B = test_df.iloc[day]['arrivals']
    obs_B = test_df.iloc[day]['converted']

    obs = [[obs_A, obs_B], [n_A, n_B]]

    stat, p, _, _  = chi2_contingency(obs)
    print(p)

And my output is (rounded to 2 decimal places):

0.89
0.88
0.83
0.57
0.63
0.41
0.43
0.83
0.96
0.99
0.98
0.97
0.94

My intuition is that even doing the calculations daily invalidates some part of the contract I made when I set up the experiment. But I'm not sure how to put this into words.

I also beleive that if I were to cahnge from a chi-square test to something like a Beta-Binomial model, and deciding if an experiment is significant via a rule like:

$$ P(\lambda_B > \lambda_A) > C $$

Where $ P(\lambda_B > \lambda_A) $ is the posterior from the Beta-Binomial and $ C $ is some fixed threshold like 0.95.

I also belive that this set up would have the same problems as the chi-square. Namely that I should only calculate once, irrespective of whether I stop based off the peeking.

Answer 1

I have not read your code but will try an answer based on the text only.

In the case of a null hypothesis $H_0$ being true, the $p$ value is just a random variable uniformely distributed between 0 and 1. If $H_0$ is true and you look often at that random variable, chances increase, that once $p < .05$. As such, you increase the cummulated alpha error of your experiment.

However, computing a $p$ value without making a decision based on it does not lead to a decision and thus does not lead to an error. In NHST the alpha error is an error in a decision based on a $p$ value. If you don't draw decisions from your $p$ value, you then cannot make a false decision aka error. So if you determine to make a decision based on one predefined $p$ value at the end then there is no alpha accumulation.

However, you should ask yourself, why do you want to compute $p$ values if not for drawing conclusions from them? The only point in computing $p$ values that I am aware of is making decisions. So whilst there is no harm in computing values that are without any practical importance, is there a point in doing so?

Answer 2

If the daily calculations don't affect the results of the test, then they don't affect the validity of the test.

However, even if those particular daily results didn't cause you to take a different action, if there were results that could have caused you to take a different action, then the validity of the test is affected. If there's going to be anyone looking at your conduct during the test to see whether the test was valid, you should avoid this sort of behavior, just to avoid the appearance of impropriety.