I am taking part in a competition right now. I know it is my job to do that well, but maybe somebody wants to discuss my problem and its solution here as this could be helfull for others in their field too.
I have trained an xgboost model (a tree based model and a linear one and an ensemble of the two). As discussed already here the mean absolute error (MAE) on the training set (where I did cross-validation) was small (approx. 0.3) then on the held-out test set the error was around 2.4. Then the competition started and the error was around 8 (!) and surprisingly the forecast was always approx 8-9 above the true value !! See the region circled in yellow in the picture:
I have to say that period of the training data ended in Oct '15 and the competition started right now (April '16 with a test period of approx 2 weeks in March).
Today I just substracted the constant values of 9 from my forecast and the error went down to 2 and I got number 3 on the leadboard (for this one-day). ;) This is the part right of the yellow line.
So what I would like to discuss:
- How does xgboost react to adding an intercept term to the model equation? Could this lead to bias if the system changes too much (as it did in my case from Oct 15 to April 16)?
- Could an xgboost model without intercept be more robust to parallel shifts in the target value?
I will go on subtracting my bias of 9 and if anybody is interested I could show you the result. It would just be more interesting to get more insight here.