statistical significance of A B test if some observation are removed due to errors

Question

I have ran an A/B test and using chi-square test of independence, results shows that group A improves on group B by less than 5% (statistically significant) YAY!!! Unfortunately, after few checks I have noticed that there were some collection errors and I had to remove some rows from the experiment. This accounts for less than 10% of the data, and fortunately the test is sill significant. However, errors did not happened at random, but always happened on a specific condition (specific day of the week). Does this means that my results are not valid? how do I quantify the error? how do I account for it in my results? I have run some test to check the difference of population between the original results and the results removed, it seems that for the variable 'days of the week' the population is different, but numbers are small, so it may not be significant.

Answer 1

That is an interesting take because the corresponding points are indeed "Missing at Random" (MAR), i.e. the missingness is due to something happening on a particular day of the week. Of course, this is not fully testable but from is described it seems perfectly reasonable. We need to control for this; before continuing here is a simplified example of why this is a problem: If the "Friday night" users are the least like to buy our services (e.g. shop at a furniture shop) and Friday night is the time of the week our system dropped, our metric will look biased due to selection bias. i.e. if we simply said: "Let's drop Friday night users from both arms and analyse the remaining data.", we would have an overoptimistic estimate as our "worse customers" would be excluded.

(Partial) Solution: Do not use a $\chi^2$ test but rather run a full (logistic) regression where the time of the week as well as the A/B group membership are used as explanatory variables - both set of variables will have to be included, buy/no buy is our response variable. If indeed there is a weekly trend this formulation should allow us account for it. To that extend, using a regression model would let us to control for other covariates as well (e.g. age) - this is optional; their inclusion if they are relevant should increase the precision of our estimate. Usual ways of computing confidence (or prediction) intervals can then be used to quantify the effect of A/B group membership. CV.SE has some very good threads on this I would recommend looking into the threads: "Why is there a difference between manually calculating a logistic regression 95% confidence interval, and using the confint() function in R?" and "Why do my p-values differ between logistic regression output, chi-squared test, and the confidence interval for the OR?" they both very instructive on the matter. The corresponding confidence will allows us to visualise the results directly. Side-note: For future reference, we also might want to initialise our testing first with an A/A test (i.e. we just record how the two randomly assigned groups behave in the absence of any treatment) so we have an idea of the inherit variability and probably catch such collection errors before our actual A/B phase.

(Full) Solution: Repeat the experiment. This is a bit heavy-handed but realistically the experiment is contaminated. Re-running the A/B test, if possible, will alleviate concerns about the influence of this kind of collection errors in the data.

Answer 2

you should include days of the week into your contingency table. This would make it a three way table, with observed counts $O_{grd}$ for group $g$, result $r$, and day of the week $d$.

Now if your model/assumption is that day of the week "doesn't matter" for your analysis. You could check this by running a separate tests for each day, where the expected values are $$E_{grd}=\frac{O_{g\bullet d}O_{\bullet r d}}{O_{\bullet\bullet d}}$$

This test allows for the treatment effect to vary by the day of the week. If you are assuming a constant treatment effect across the days then your expected values are $$E_{grd}=\frac{O_{g\bullet \bullet}O_{\bullet r \bullet}O_{\bullet\bullet d}}{O^2_{\bullet\bullet\bullet}}$$

These can both be done using logistic regression, and by inspecting either the "deviance" or "pearson" residuals, you can get your chi square test. however it can be useful to work through the observed/expected counts to see where the assumptions are failing