If it is a regression model (objective can be reg:squarederror), then the leaf value is the prediction of that tree for the given data point. The leaf value can be negative based on your target variable. The final prediction for that data point will be sum of leaf values in all the trees for that point.
If it is a classification model (objective can be ...
I dont understand how this can be overfitting. For me overfitting occurs when you cannot generalize anymore. Here you test-rmse keeps decreasing which means that you have not overfitted yet. Increase the capacity of the model and increase the boosting rounds until you have seen test-rmse decrease and then increase. This is the behaviour you should spot
[documentation of XGBoost] provides a great introduction to the boosted trees algorithm.
Perhaps, I can help out answering your first question:
I think you state two false alternatives. The algorithm uses the entire training set (leaving aside the fact that you can sample observations in ...
Because each tree grown by xgboost splits y into leaf nodes conditional on X. The prediction (or leaf weight) is the value which minimises the loss function for the split y's - so for example if the loss function is squared error, the leaf weight is the mean of the y's.
So say you have 1 tree of depth 1 (i.e. a single split on 1 feature). All this model ...
It seems to me that tree-based models are very bad at extrapolation, please look at this discussion https://www.kaggle.com/c/web-traffic-time-series-forecasting/discussion/38352.
Some people also pointed out that XGBoost has some weak "potential" at extrapolation https://github.com/dmlc/xgboost/issues/1581, but in general and in my personal applications, I ...
Update: The R2 function is deprecated.
You can get R2 by the postResample function now.
The code is here and shows in line 136 and 142 the formula:
resamplCor <- try(cor(pred, obs, use = "pairwise.complete.obs"), silent = TRUE
out <- c(sqrt(mse), resamplCor^2, mae)
So postResample uses the squared correlation to calculate R2.