Model performance in time-series forecasting with some outliers

Question

I'm creating forecasts for products where some of them have large seasonal spike during times like Christmas and/or Easter but relatively low sales volume on other times. For this particular product shown in the graph below, the most important part is the spike during Christmas.

So the model that is better at predicting the spike is considered to be the best model. I need to be able to programmatically pick Prophet here since it is better than XGBoost in predicting the spike. When using error metrics like MAE, RMSE and MASE, XGBoost is considered to be the most accurate model but it misses the spike.

What options do I have? Could a different error metric help here? Or could I make changes to XGBoost to better handle a time series like this, e.g. selecting different learning objective (I'm using the default, squarederror)?

Answer 1

One possible approach, if detecting sudden "spikes" is more important one could use difference of consecutive time points in the Out of Sample (OOS) test in the performance metric.

Let's say OOS is $y_{k}$ to $y_{k+m}$ with estimated values with hat then $\Delta MAE$ would read

$$ \Delta MAE = \frac{1}{m} \sum_{j}^{m-1} | \Delta{y_{j}} - \Delta{ \hat{y}_{j}} | $$

This will penalise models that are not doing well in high differences, where $\Delta{y_{j}} = |y_{j} - y_{j+1}| $ and $\Delta{\hat{y}_{j}} = |\hat{y}_{j} - \hat{y}_{j+1}| $ are absolute differences. This won't guarantee, but would reduce the bias due to high differences on other metrics.

However, more robust approach would be detecting outliers, seperately and also testing for non-stationarity could help.