I have been asked a test about A/B testing that I'm unsure how to answer.

Say you're running a conversion rate experiment, and my A variation has a conversion rate of 2% and my B variation has a 2.2% conversion rate (so a 10% relative change).

I set up my system to collect data daily, and I calculate the p-value from a chi-squared test daily. However, depite peeking at the p-value daily, I don't stop the experiment until the agreed stopping point (14 days after the experiment started).

My question is: does this affect things like the statistical power/false positive rate?

I simulated the process with the code below:

import pandas as pd
import numpy as np
import dsutil as hfds
from pymc3.stats import hpd
from scipy.special import betaln
from scipy.stats import beta
from scipy.stats import chi2_contingency

class DGP(object):
    Data Generating Process

    def __init__(self, arrivals_mean=100, 
        self.arrivals_mean = arrivals_mean
        self.conversion_rate = conversion_rate
        self.n_days = n_days
        self.n_reps = n_reps
    def generate_data(self, mde=0.0):
        Simulate a conversion experiment with Poisson arrivals and a conversion_rate conversion probability
        n_arrivals = np.random.poisson(lam=self.arrivals_mean, size=self.n_days*self.n_reps)
        n_converted = np.random.binomial(n_arrivals, self.conversion_rate+(self.conversion_rate*mde))
        iterables = [range(self.n_reps), range(self.n_days)]
        index = pd.MultiIndex.from_product(iterables, names=['replicate', 'day'])
        df = pd.DataFrame(data={'arrivals': n_arrivals, 'converted': n_converted}, index=index)
        return df.cumsum()
    def simulate(self):
        control_df = self.generate_data()
        test_df = self.generate_data(self.mde)
        return control_df, test_df

n_trials = 1
days = 13
positives = 0

control_df, test_df = DGP(mde=0.02).simulate()
for day in range(days): 
    n_A = control_df.iloc[day]['arrivals']
    obs_A = control_df.iloc[day]['converted']

    n_B = test_df.iloc[day]['arrivals']
    obs_B = test_df.iloc[day]['converted']

    obs = [[obs_A, obs_B], [n_A, n_B]]

    stat, p, _, _  = chi2_contingency(obs)

And my output is (rounded to 2 decimal places):


My intuition is that even doing the calculations daily invalidates some part of the contract I made when I set up the experiment. But I'm not sure how to put this into words.

I also beleive that if I were to cahnge from a chi-square test to something like a Beta-Binomial model, and deciding if an experiment is significant via a rule like:

$$ P(\lambda_B > \lambda_A) > C $$

Where $ P(\lambda_B > \lambda_A) $ is the posterior from the Beta-Binomial and $ C $ is some fixed threshold like 0.95.

I also belive that this set up would have the same problems as the chi-square. Namely that I should only calculate once, irrespective of whether I stop based off the peeking.


2 Answers 2


I have not read your code but will try an answer based on the text only.

In the case of a null hypothesis $H_0$ being true, the $p$ value is just a random variable uniformely distributed between 0 and 1. If $H_0$ is true and you look often at that random variable, chances increase, that once $p < .05$. As such, you increase the cummulated alpha error of your experiment.

However, computing a $p$ value without making a decision based on it does not lead to a decision and thus does not lead to an error. In NHST the alpha error is an error in a decision based on a $p$ value. If you don't draw decisions from your $p$ value, you then cannot make a false decision aka error. So if you determine to make a decision based on one predefined $p$ value at the end then there is no alpha accumulation.

However, you should ask yourself, why do you want to compute $p$ values if not for drawing conclusions from them? The only point in computing $p$ values that I am aware of is making decisions. So whilst there is no harm in computing values that are without any practical importance, is there a point in doing so?

  • 3
    $\begingroup$ Specifically - I have an opaque piece of software that's hard to refactor that's calculating p values daily. I just wanted to know if I disabled the UI element showing the p-value until the experiment is over, then that p-value will be fine to make decisions off. $\endgroup$
    – Tom Kealy
    Nov 17, 2020 at 11:31
  • 1
    $\begingroup$ @Bernhard The only point in computing p values that I am aware of is making decisions. P-values are useful also to rank results when the same test is applied to different experiments sharing the same setup. An example is ranking genes in a differential gene expression study where you test several genes on the same subjects under the same conditions. $\endgroup$
    – dariober
    Nov 17, 2020 at 12:02
  • $\begingroup$ @dariober Yeah, but what's the point of doing that if no actions will be affected by it? $\endgroup$ Nov 17, 2020 at 19:59
  • $\begingroup$ @Acccumulation Sure, what I meant is that p-values are not only for making binary decisions like accept/reject. You can rank genes based on p-values without deciding which genes are "significant". Then, of course, ranking is in itself a decision with effects. $\endgroup$
    – dariober
    Nov 18, 2020 at 9:35
  • 1
    $\begingroup$ @dariober A $p$ value as such is only defined in the case of $H_0$ being true. So it is only defined when it is uniformely distributed and thus worthless. Obviously in case of any specificly given test there is defined behaviour in the case of $H_0$ not being true. I can see how such a $p$ computed by a certain test under some fixed other conditions can be used for comparisons. It still feels odd, like there should be better (more elegant?) measures for that purpose but I am neither a mathematician nor a geneticist who could tell. Thank you for pointing us to that special use case. $\endgroup$
    – Bernhard
    Nov 18, 2020 at 10:03

If the daily calculations don't affect the results of the test, then they don't affect the validity of the test.

However, even if those particular daily results didn't cause you to take a different action, if there were results that could have caused you to take a different action, then the validity of the test is affected. If there's going to be anyone looking at your conduct during the test to see whether the test was valid, you should avoid this sort of behavior, just to avoid the appearance of impropriety.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.