I am wondering if, instead of "paying for" an A/B test for all hypothesis' (designing an experiment, running it under controlled situation etc), a statistical analysis can be conducted on past data as a "first pass" on a hypothesis. If it shows there is evidence of the hypothesis being true, the A/B test can be designed, and if not you don't need to waste your money.

I cannot seem to find a "full" example of the counterfactual theory (please see reference below) in practice using actual data (for example the aspirin example is interesting but it would be good to see a data set and then what was done in R to derive a conclusion) which may answer this question.

Does anyone have anything like this / can link me to further reading / can share a more concrete example and verify this method does indeed act as a first-pass for an AB test (or have I misunderstood the notes)?

Thank you in advance!

Reference: http://www.stats.ox.ac.uk/~evans/Counterfactuals/LectureNotes.pdf

Note: this is not homework I use this technique a lot at work therefore am curious.

  • $\begingroup$ Quick answer/comment: What you describe is reasonable. Nobody designs an A/B test out of thin air. Having said that, to draw anything more than "educated guesses", you have to accept that the baseline bias across the past data is zero as well as that the differential treatment effect bias is zero too. These are very demanding assumptions for an observational study. You could put all your eggs on the propensity-score basket and hope for the best but I would advise it. I am reading the book "Counterfactuals and Causal Inference" by Morgan & Winship; it is very clear, I would recommend it. $\endgroup$ – usεr11852 Jul 14 '17 at 0:05
  • $\begingroup$ Thanks! I agree but a lot of what I see going on here is "I think this should be an A/B test so let's run it" so I'm trying to filter the good ones vs bad ones. I had a read of that book before finding these notes, but found it very theoretical, and wasn't sure how to apply it day-to-day with real data (hence the link above, which was the closest I could find to real data) with explanation and conclusions. I understand the stats wont be perfect but act as a first-pass. At the moment we are not even doing that! $\endgroup$ – Dino Abraham Jul 14 '17 at 0:17
  • $\begingroup$ Look into these two papers. It'll turn your testing world upside down and inside out. I've been meaning to do a series of articles condensing these into easier to read form and going through one concept at a time, but probably won't get to it for a while. You're after the second one with this question, but it would be helpful to read the first... first. Toward evidence-based medical statistics. 1: The P value fallacy. Toward evidence-based medical statistics. 2: The Bayes factor. $\endgroup$ – Alex Firsov Jul 14 '17 at 1:26
  • $\begingroup$ Thanks @AlexFirsov - do you happen to have a link to the full paper or have an example? I can't seem to access the second link. $\endgroup$ – Dino Abraham Jul 16 '17 at 10:58
  • $\begingroup$ I found this link by googling the paper's name in quotes followed by "pdf" $\endgroup$ – Alex Firsov Jul 16 '17 at 15:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.