Say I have a target metric and a second metric related to the target, and I want to measure the effect of an intervention on the target metric. The intervention affects both metrics, but not always equally. I have data for n days before and after the intervention. The target metric has higher variance than the related metric, but the related metric only has correlation of ~0.5 to the target metric. After the intervention, if I want the effect estimate with the lowest MAPE, should I measure the effect using the target metric or the related metric?

I could not find any literature on this common problem. I attempted the problem myself, using simulation to produce time series similar to the given data, and adding a percentage increase to the target and related metric. Then I would use DID to make an estimate of the increase, directly if using the target, or estimating the target using regression on the related metric and applying DID. Defining the increase that should be added to the related metric is difficult, as the increase must reflect not only the relationship between them, but also how likely it is that an intervention will change the relationship. I chose to add the same percentage increase to both target and related metric, with a standard error depending on the time-varying correlation between them. This produced reasonable results (for low sample sizes, related metric has lowest MAPE, then at some point target becomes better), but my methods were somewhat arbitrary.

1. What existing research/methods are there on measuring effect when multiple metrics are available?

2. What are some alternate approaches you think could work?

Also, I have seen Granger causality and Vector Autoregression referenced in similar questions. Could these be of use?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.