I'm creating forecasts for products where some of them have large seasonal spike during times like Christmas and/or Easter but relatively low sales volume on other times. For this particular product shown in the graph below, the most important part is the spike during Christmas.

So the model that is better at predicting the spike is considered to be the best model. I need to be able to programmatically pick Prophet here since it is better than XGBoost in predicting the spike. When using error metrics like MAE, RMSE and MASE, XGBoost is considered to be the most accurate model but it misses the spike.

What options do I have? Could a different error metric help here? Or could I make changes to XGBoost to better handle a time series like this, e.g. selecting different learning objective (I'm using the default, squarederror)?

enter image description here


1 Answer 1


One possible approach, if detecting sudden "spikes" is more important one could use difference of consecutive time points in the Out of Sample (OOS) test in the performance metric.

Let's say OOS is $y_{k}$ to $y_{k+m}$ with estimated values with hat then $\Delta MAE$ would read

$$ \Delta MAE = \frac{1}{m} \sum_{j}^{m-1} | \Delta{y_{j}} - \Delta{ \hat{y}_{j}} | $$

This will penalise models that are not doing well in high differences, where $\Delta{y_{j}} = |y_{j} - y_{j+1}| $ and $\Delta{\hat{y}_{j}} = |\hat{y}_{j} - \hat{y}_{j+1}| $ are absolute differences. This won't guarantee, but would reduce the bias due to high differences on other metrics.

However, more robust approach would be detecting outliers, seperately and also testing for non-stationarity could help.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.