Statistical test: Does actual time series data deviate from forecast?

Question

I have made a prediction of future sales based on an ARIMA model. The ARIMA model is based on past data, during which there has been no marketing activity. During the period predicted by ARIMA, I will be running an Online marketing campaign.

I would like to evaluate whether the marketing campaign had a statistically significant impact. I.e. I would like to test: H0: actual mean sales = ARIMA predicted mean sales (marketing did not work), H1: actual mean sales > ARIMA predicted mean sales.

What is the best way to solve for this?

Is it as simple as running a one-tail t-test (Welch's t-test if there is a big difference in variance between the two series?)

Would really appreciate any help with this! Thanks

Answer 1

is it as simple as running a one-tail t-test to compare the means of the predicted time series vs. the actual time series

The big problem is that with time series you don't have independence (after all, that's why you have an ARIMA model!), which the t-test definitely relies on; in general the forecasts would also be expected to be correlated (cross-correlated at lag 0) with the actual.

Welch's t-test if there is a big difference in variance between the two series?

No; the data would have to be regarded as paired, so you'd be doing a one-sample test on the pairwise differences, but you still need to account for serial dependence since it would impact the type I error rate.