We had a change in our production code which we did not perform A/B test for. A few days after the product is changed, we have been seeing a significant drop in our quality metrics.

Is there any way to causally contribute the sudden drop to the change in the production code as we don't have any counterfactual.

Also, how would you go about analyzing the results to demonstrate the drop can be "correlated" with the change in the code? I checked different segments of customers and they all seem to have a similar drop.

Any idea is highly appreciated.

  • 1
    $\begingroup$ do you have a control group meaning do you have customers that were not exposed to change in code/program? $\endgroup$ – forecaster Oct 1 at 14:54
  • $\begingroup$ Proving causality with or without statistical analysis can be difficult. But one has the right to be suspicious. To what else might you attribute the unanticipated decrease in quality? (Change is suppliers of raw materials? Decrease in employee morale? Change in the weather?) $\endgroup$ – BruceET Oct 1 at 18:41
  • 1
    $\begingroup$ The best proof is if you can go back to the old code and make the problem go away (for a statistically significant sample). That rules out alot of other potential causes. $\endgroup$ – MikeP Oct 1 at 18:46
  • $\begingroup$ we don't have any customer left in the old code. Any other way of showing that users' behavior has changed after this exposure? What type of user-level analysis can we perform? $\endgroup$ – aghd Oct 2 at 3:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.