I have a dataset on which there are about 1000 dummy variables indicating location. I do not have access to lat/long.
I am using xgboost to train it. The more that I train it does seem to be lowering the test-set error. However, when I try to look at the branches of the trees using
trees = xgb.model.dt.tree(names,model = model2) all values of the split (
trees$Split) are a negative value (-1.00136e-05).
Considering that these are dummy variables (1 or 0) for a negative split value it will always go to the yes branch. My question is how is it even learning anything with a negative split value? It seems the more rounds that I let it train the better the algorithm gets in terms of test set error. This doesn't make much sense.
If it helps its a multi-class classification problem with 3 classes and the error metric is log-loss.
edit: to answer questions of standardisation and -1/1 see below:
 1 0
edit 2: This number seems to occur even on his basic example here
edit 3: Minimal example included:
require(xgboost) set.seed(1) data(agaricus.train, package='xgboost') train <- agaricus.train bst <- xgboost(data = as.matrix(train$data), label = train$label, max.depth = 2, eta = 1, nround = 2, nthread = 2, objective = "binary:logistic") trees = xgb.model.dt.tree(dimnames(train$data)[],model = bst) print(head(trees$Split)) >>>  "-1.00136e-05" "-1.00136e-05" "-1.00136e-05" NA NA NA
As Vadim points out below I was using a old version of xgboost (the one offered on cran). Use the following to update the version and you will get 0.5 as expected.
install.packages("drat", repos="https://cran.rstudio.com") drat:::addRepo("dmlc") install.packages("xgboost", repos="http://dmlc.ml/drat/", type = "source")