We had a change in our production code which we did not perform A/B test for. A few days after the product is changed, we have been seeing a significant drop in our quality metrics.
Is there any way to causally contribute the sudden drop to the change in the production code as we don't have any counterfactual.
Also, how would you go about analyzing the results to demonstrate the drop can be "correlated" with the change in the code? I checked different segments of customers and they all seem to have a similar drop.
Any idea is highly appreciated.