The following example borrow from forecastxgb author's blog, the tree-based model can't extrapolate in it's nature, but there are definitely some method to combine the benefit of tree model (interaction factors) and linear model's trend extrapolate ability. Could anyone give some ideas?

I have seen some kaggle solution, some people advise using the linear model prediction as the tree model's feature, it can imporve the prediction result, but how to improve the extrapolate?

Another idea is using the xgboost predict the residual of the linear model, this can help the prediction a lot.

Is there anyway?

library(xgboost)  # extreme gradient boosting
set.seed(134) # for reproducibility
x <- 1:100 + rnorm(100)
y <-   3 + 0.3 * x + rnorm(100)
extrap <- data.frame(x = 101:120 + rnorm(20))

xg_params <- list(objective = "reg:linear", max.depth = 2)
mod_cv <- xgb.cv(label = y, params = xg_params, data = as.matrix(x), nrounds = 40, nfold = 10) 
# choose nrounds that gives best value of root mean square error on the training set
best_nrounds <- which(mod_cv$evaluation_log$test_rmse_mean == min(mod_cv$evaluation_log$test_rmse_mean))
mod_xg <- xgboost(label = y, params = xg_params, data = as.matrix(x), nrounds = best_nrounds)

p <- function(title){
  plot(x, y, xlim = c(0, 150), ylim = c(0, 50), pch = 19, cex = 0.6,
      main = title, xlab = "", ylab = "", font.main = 1)

predshape <- 1
p("Extreme gradient boosting")
points(extrap$x, predict(mod_xg, newdata = as.matrix(extrap)), col = "darkgreen", pch = predshape)

xgboost forcasting result

mod_lm <- lm(y ~ x)
p("Linear regression")
points(extrap$x, predict(mod_lm, newdata = extrap), col = "red", pch = predshape)

linear model forecasting result

  • $\begingroup$ In this case, I have tried use the prediction of linear model as the xgboost feature, but it didn't help the result (is it true?). I also tried use the residual of linear model as the xgboost feature, it seem better than the former result, but still have bad extrapolate. The example come from this site:ellisp.github.io/blog/2016/12/10/extrapolation $\endgroup$ – wolfe Mar 23 '17 at 23:18
  • $\begingroup$ I have read a lot of question about the extrapolate problem of tree-based model, but it seems there are not solution about it. $\endgroup$ – wolfe Mar 23 '17 at 23:30
  • $\begingroup$ I haven't enough reputation to invite @MatthewDrury to answer my question, I think he can answer it. His answer in another correlation question:stats.stackexchange.com/questions/262114/… $\endgroup$ – wolfe Mar 24 '17 at 5:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.