Measuring events in sales time series I am trying to measure the effect of an event on some sales data. An event might be turning on a new payment method for customers, or a discount coupon promotion. Does it make sense to try a time series decomposition into Trend-Seasonal-Residual, and then try to spot the effect in the Residual component?
Maybe it would make sense to do this for the period before the event, and then compare a forecast over the event period to the actual sales.
If either of these methods make sense, how might I measure the significance of any findings?
Are there other methods to check for an effect and to evaluate the significance of my findings?
 A: What you normally do in such situations is to apply randomized A/B testing. I.e., you randomly separate your customers into two parts $g_A$ (the control group) and $g_B$ (the treatment group), and then "treat" one group $g_B$ with the event like a discount coupon promotion, while not changing anything for the other group $g_A$. Then, after you think the treatment should have come into effect, you compare the sales data $sale(g_A, t)$ versus $sale(g_B, t)$. E.g., you could take the difference. You do this by comparing the sales data for both groups at the same times $t$, i.e. you choose some points $t_i, i=1,\ldots,n$ in time and then compute $sale(g_B, t_i) - sale(g_A, t_i)$ for $i=1,\ldots,n$. Thus, with this approach, you don't learn any time series model.
However, sometimes it is not possible to partition your customers into treatment and control groups. E.g., maybe you cannot use two payment methods in parallel. In this case, you could use the approach you mentioned in your question, exploiting the time series character of your data. You first learn a time series model on your sales data, before you change anything. Then, after you are confident that the model works sufficiently well, you implement the change and again record the sales data. Finally, you compare the actual sales data with the predicted sales. If you want to follow this approach, you could e.g. use Google's CausalImpact. This software also provides for extensions like using additional covariates.
If you want to know whether the changes are significant, you could use a paired t-test. If the assumptions of t-tests are not satisfied in your case, you should consider more generally applicable alternatives.
