I have ran an A/B test and using chi-square test of independence, results shows that group A improves on group B by less than 5% (statistically significant) YAY!!! Unfortunately, after few checks I have noticed that there were some collection errors and I had to remove some rows from the experiment. This accounts for less than 10% of the data, and fortunately the test is sill significant. However, errors did not happened at random, but always happened on a specific condition (specific day of the week). Does this means that my results are not valid? how do I quantify the error? how do I account for it in my results? I have run some test to check the difference of population between the original results and the results removed, it seems that for the variable 'days of the week' the population is different, but numbers are small, so it may not be significant.

  • $\begingroup$ +1 Good on you to catch such a problem in your data collection procedure and then question its influence. Some people would just gloss over this. (+1) $\endgroup$ – usεr11852 says Reinstate Monic Mar 31 '19 at 12:06

That is an interesting take because the corresponding points are indeed "Missing at Random" (MAR), i.e. the missingness is due to something happening on a particular day of the week. Of course, this is not fully testable but from is described it seems perfectly reasonable. We need to control for this; before continuing here is a simplified example of why this is a problem: If the "Friday night" users are the least like to buy our services (e.g. shop at a furniture shop) and Friday night is the time of the week our system dropped, our metric will look biased due to selection bias. i.e. if we simply said: "Let's drop Friday night users from both arms and analyse the remaining data.", we would have an overoptimistic estimate as our "worse customers" would be excluded.

(Partial) Solution: Do not use a $\chi^2$ test but rather run a full (logistic) regression where the time of the week as well as the A/B group membership are used as explanatory variables - both set of variables will have to be included, buy/no buy is our response variable. If indeed there is a weekly trend this formulation should allow us account for it. To that extend, using a regression model would let us to control for other covariates as well (e.g. age) - this is optional; their inclusion if they are relevant should increase the precision of our estimate. Usual ways of computing confidence (or prediction) intervals can then be used to quantify the effect of A/B group membership. CV.SE has some very good threads on this I would recommend looking into the threads: "Why is there a difference between manually calculating a logistic regression 95% confidence interval, and using the confint() function in R?" and "Why do my p-values differ between logistic regression output, chi-squared test, and the confidence interval for the OR?" they both very instructive on the matter. The corresponding confidence will allows us to visualise the results directly. Side-note: For future reference, we also might want to initialise our testing first with an A/A test (i.e. we just record how the two randomly assigned groups behave in the absence of any treatment) so we have an idea of the inherit variability and probably catch such collection errors before our actual A/B phase.

(Full) Solution: Repeat the experiment. This is a bit heavy-handed but realistically the experiment is contaminated. Re-running the A/B test, if possible, will alleviate concerns about the influence of this kind of collection errors in the data.

  • $\begingroup$ Hi, and thanks a lot for your answer. In your 'partial' solution, what would be the response variable? still my original one? buy or did not buy? The input features will be time of the week and group only or including all the variables? am I just adding the group to the input variables? in which way this can allow us to account for it? can you expand? also, can you expand on how I would quantify the effect of the group and the error and then present the results? thank you $\endgroup$ – DarioB Apr 2 '19 at 10:48
  • $\begingroup$ I am glad I can help. You have the right idea; yes, the response variable would be if some buys or not (that's why I mentioned logistic) and yes, you will include both time of the week as well as group membership as response. I added some references on how to calculate confidence intervals for logistic regression and some other small comments too. $\endgroup$ – usεr11852 says Reinstate Monic Apr 2 '19 at 22:12
  • $\begingroup$ hi, thanks a lot for this, unfortunately I am still struggling to understand, I am a real noob on this. Firstly, do you maybe meant that both time of the week as well as group membership should be included as input features (rather than response variable)? to recap: logistic regression to predict 'buy or not buy', input features would be all my previous (day of the week, age, sex) plus group membership? I have read your links but still struggling to understand how I would quantify the effect of my sampling error, can you please explain further? thank you so much. $\endgroup$ – DarioB Apr 21 '19 at 13:46
  • $\begingroup$ No biggie. I will expand on this tonight. $\endgroup$ – usεr11852 says Reinstate Monic Apr 21 '19 at 15:52
  • $\begingroup$ I am glad I could help. For the follow-up question: Yes, your understanding is correct, I meant that "both time of the week as well as group membership should be included as input features". The standard way of assessing sampling variability is through resampling techniques like bootstrapping our sample. $\endgroup$ – usεr11852 says Reinstate Monic Apr 21 '19 at 17:26

you should include days of the week into your contingency table. This would make it a three way table, with observed counts $O_{grd}$ for group $g$, result $r$, and day of the week $d$.

Now if your model/assumption is that day of the week "doesn't matter" for your analysis. You could check this by running a separate tests for each day, where the expected values are $$E_{grd}=\frac{O_{g\bullet d}O_{\bullet r d}}{O_{\bullet\bullet d}}$$

This test allows for the treatment effect to vary by the day of the week. If you are assuming a constant treatment effect across the days then your expected values are $$E_{grd}=\frac{O_{g\bullet \bullet}O_{\bullet r \bullet}O_{\bullet\bullet d}}{O^2_{\bullet\bullet\bullet}}$$

These can both be done using logistic regression, and by inspecting either the "deviance" or "pearson" residuals, you can get your chi square test. however it can be useful to work through the observed/expected counts to see where the assumptions are failing

  • $\begingroup$ If indeed the OP runs separate tests for each day, shouldn't they do some minor multiple testing corrections? (I do consider three-way (or more) contingency tables a bit "self-defeating", if we open the door to multiple factors we might as well go with a regression approach, but OK, I appreciate this is not a universal view.) $\endgroup$ – usεr11852 says Reinstate Monic Apr 7 '19 at 11:48
  • $\begingroup$ Hi, not sure if I understand fully the procedure, but if my data already has a difference in sales at weekends (hence not constant across the day of the week), I can't use this approach right? because that alone will produce a significant different in my contingency table result, and I won't be able to understand if that was produced by my sampling error. Is that correct? $\endgroup$ – DarioB Apr 21 '19 at 13:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.