8
$\begingroup$

I am taking part in a competition right now. I know it is my job to do that well, but maybe somebody wants to discuss my problem and its solution here as this could be helfull for others in their field too.

I have trained an xgboost model (a tree based model and a linear one and an ensemble of the two). As discussed already here the mean absolute error (MAE) on the training set (where I did cross-validation) was small (approx. 0.3) then on the held-out test set the error was around 2.4. Then the competition started and the error was around 8 (!) and surprisingly the forecast was always approx 8-9 above the true value !! See the region circled in yellow in the picture:

enter image description here

I have to say that period of the training data ended in Oct '15 and the competition started right now (April '16 with a test period of approx 2 weeks in March).

Today I just substracted the constant values of 9 from my forecast and the error went down to 2 and I got number 3 on the leadboard (for this one-day). ;) This is the part right of the yellow line.

So what I would like to discuss:

  • How does xgboost react to adding an intercept term to the model equation? Could this lead to bias if the system changes too much (as it did in my case from Oct 15 to April 16)?
  • Could an xgboost model without intercept be more robust to parallel shifts in the target value?

I will go on subtracting my bias of 9 and if anybody is interested I could show you the result. It would just be more interesting to get more insight here.

$\endgroup$
  • $\begingroup$ Sounds like you manually changed your model based on test data, so yes it is better, but it is not reproducible. Your model is matching the curvature of the data pretty well, the reason for the error in this region seems to be at the start, when the red line plummets while the blue line rises. I'd try to find how to model this behavior. $\endgroup$ – Winks Apr 7 '16 at 9:03
  • $\begingroup$ @Winks thanks for coming back to me! I have to say that there was a test period before the competiation and even there there was the 8-9 error and always positive ... so it is not just the move in the beginning of the screenshot but appearantly the whole system has changed. Other competitors seem to have it right right from the start ... so yes, maybe I am just to bad .. or they use better data. I was just suprised to see this bad error now while everything was that robust in on the training data (and train/test split and x-validation ...). $\endgroup$ – Ric Apr 7 '16 at 9:10
3
$\begingroup$

I will answer myself and let you know my findings in case anybody is interested.

First the bias: I took the time to collect all the recent data and format it correclty and so on. I should have done this long before. The picture is the following:

enter image description here

You see the data from the end of 2015 and then April 16. The price level is totally different. A model trained on 2015 data can in no way get this change.

Second: The fit of xgboost. I really liked the following set-up. train and test error are much close now and still good:

xgb_grid_1 <- expand.grid(
    nrounds = c(12000),
    eta = c(0.01),
    max_depth = c(3),
    gamma = 1,
    colsample_bytree = c(0.7),
    min_child_weight = c(5) 
  )

  xgb_train_1 <- train(
    x = training,y = model.data$Price[inTrain],
    trControl = ctrl,
    tuneGrid = xgb_grid_1,
    method="xgbTree" 
   ,subsample = 0.8
    )

Thus I use a lot of trees and all of them are at most 3 splits deep (as recommended here). Doing this the calculation is quick (the tree size grows by a factor of 2 with each split) and the overfit seems to be reduced.

My summary: use trees with a small number of leaves but a lot of them and look for recent data. For the competition this was bad luck for me...

$\endgroup$
  • $\begingroup$ Thank you for sharing this (+1). Just to ask the obvious regarding your code: You clearly do not search along a grid here. Just train (and gets resampling statistics) for a single parameter setting. Did you come across the "3" by using different parameter values? Was "3" optimal based on RMSE or some other criterion? $\endgroup$ – usεr11852 says Reinstate Monic Jan 4 '17 at 0:27
  • $\begingroup$ @usεr11852 In my "first days" with xgoost I chose max_depth much too big. If you follow the link in my answer ("here") then you see a discussion. Finally I just chose 3. As indicated in the code above you can use the caret package to do grid search, alternatively as I remember you can use some routines in xgboost directly and recently I started using mlr which does the same. The more noise you have the lower I choose the depth (often just 1 or 2). $\endgroup$ – Ric Jan 4 '17 at 7:25
  • $\begingroup$ Thank you for the reply but it's not what I am asking in my comment :D. I read the ("here") link prior to my comment but you did not comment what depth was the optimal one based on your criterion (MAE?) just that "I took your advice ... to 3 but ...". So was "3" optimal based on your RMSE/MAE error or what else (minimum discrepancy between test/training error)? For example I routinely grid-search the tree-depth directly but maybe you didn't but still used "3"? $\endgroup$ – usεr11852 says Reinstate Monic Jan 4 '17 at 12:47
  • $\begingroup$ Strictly comment-wise, why did you move from caret to mlr? To me they seem rather overlapping buy maybe I am missing out something else. $\endgroup$ – usεr11852 says Reinstate Monic Jan 4 '17 at 12:50
  • $\begingroup$ @usεr11852 for the first comment: if this is a ML competition then you should most probably grid-search the depth. This was a forecasting competition with a clear signal and in my mind the choice of good features (weather variables e.g) was more crucial. As the target performed a regime switch I didn't have enough representative data during the live-peroid so I just took a depth of three. $\endgroup$ – Ric Jan 4 '17 at 12:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.