I have a dataset on which there are about 1000 dummy variables indicating location. I do not have access to lat/long.

I am using xgboost to train it. The more that I train it does seem to be lowering the test-set error. However, when I try to look at the branches of the trees using trees = xgb.model.dt.tree(names,model = model2) all values of the split (trees$Split) are a negative value (-1.00136e-05).

Considering that these are dummy variables (1 or 0) for a negative split value it will always go to the yes branch. My question is how is it even learning anything with a negative split value? It seems the more rounds that I let it train the better the algorithm gets in terms of test set error. This doesn't make much sense.

If it helps its a multi-class classification problem with 3 classes and the error metric is log-loss.

edit: to answer questions of standardisation and -1/1 see below:


[1] 1 0

edit 2: This number seems to occur even on his basic example here

edit 3: Minimal example included:

data(agaricus.train, package='xgboost')
train <- agaricus.train
bst <- xgboost(data = as.matrix(train$data), label = train$label, max.depth = 2, eta = 1, nround = 2,
               nthread = 2, objective = "binary:logistic")
trees = xgb.model.dt.tree(dimnames(train$data)[[2]],model = bst)
>>> [1] "-1.00136e-05" "-1.00136e-05" "-1.00136e-05" NA NA NA   


As Vadim points out below I was using a old version of xgboost (the one offered on cran). Use the following to update the version and you will get 0.5 as expected.

install.packages("drat", repos="https://cran.rstudio.com")
install.packages("xgboost", repos="http://dmlc.ml/drat/", type = "source")

migrated from stackoverflow.com Feb 2 '16 at 9:38

This question came from our site for professional and enthusiast programmers.

  • $\begingroup$ Are you sure they aren't coded as -1, 1? $\endgroup$ – Adrian Feb 2 '16 at 10:09
  • 1
    $\begingroup$ perhaps you have a standardization somewhere $\endgroup$ – rapaio Feb 2 '16 at 11:11
  • $\begingroup$ see edits above @rapaio $\endgroup$ – sachinruk Feb 2 '16 at 23:03
  • 1
    $\begingroup$ You really can't be sure that xgboost is not internally standardizing the variables without scouring the documentation and inspecting the source code, regardless of what your data looks like in R. That said, I don't see any reason for a tree based model to standardize. $\endgroup$ – Matthew Drury Feb 2 '16 at 23:07
  • $\begingroup$ This certainly seems to be the case as it happens on the basic example when I inspect the tree. $\endgroup$ – sachinruk Feb 3 '16 at 3:25

This is most likely because your data is in sparse format. E.g., enforcing the data to be dense in the basic example (using as.matrix(train$data)) would show the splits to be at 0.5.

For sparse "dummy" data, one is the only actual value that a feature might have. Thus, the search for best split_value for such a feature doesn't really move far from its initial value of zero, mostly just the offset with rt_eps.

  • $\begingroup$ I'm afraid I've already done as.matrix to change the data frame. The data isn't sparse $\endgroup$ – sachinruk Feb 3 '16 at 6:27
  • $\begingroup$ Then please provide a reproducible example. $\endgroup$ – Vadim Khotilovich Feb 3 '16 at 7:02
  • $\begingroup$ As requested minimal example included $\endgroup$ – sachinruk Feb 3 '16 at 11:27
  • 1
    $\begingroup$ This is the "dense matrix" from the "basic example". And I get the [1] "0.5" "0.5" "0.5" NA NA NA result from it using either the most recent or the pre-"brick" version of xgboost code on 64bit windows and linux. What xgboost code and what system are you running? $\endgroup$ – Vadim Khotilovich Feb 3 '16 at 16:44
  • $\begingroup$ So it turns out that I had to update the xgboost via devtools and NOT the cran version. $\endgroup$ – sachinruk Feb 4 '16 at 0:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.