2
$\begingroup$

I have a dataset on which there are about 1000 dummy variables indicating location. I do not have access to lat/long.

I am using xgboost to train it. The more that I train it does seem to be lowering the test-set error. However, when I try to look at the branches of the trees using trees = xgb.model.dt.tree(names,model = model2) all values of the split (trees$Split) are a negative value (-1.00136e-05).

Considering that these are dummy variables (1 or 0) for a negative split value it will always go to the yes branch. My question is how is it even learning anything with a negative split value? It seems the more rounds that I let it train the better the algorithm gets in terms of test set error. This doesn't make much sense.

If it helps its a multi-class classification problem with 3 classes and the error metric is log-loss.

edit: to answer questions of standardisation and -1/1 see below:

unique(as.vector(as.matrix(train_set)))

[1] 1 0

edit 2: This number seems to occur even on his basic example here

edit 3: Minimal example included:

require(xgboost)
set.seed(1)
data(agaricus.train, package='xgboost')
train <- agaricus.train
bst <- xgboost(data = as.matrix(train$data), label = train$label, max.depth = 2, eta = 1, nround = 2,
               nthread = 2, objective = "binary:logistic")
trees = xgb.model.dt.tree(dimnames(train$data)[[2]],model = bst)
    print(head(trees$Split))
>>> [1] "-1.00136e-05" "-1.00136e-05" "-1.00136e-05" NA NA NA   

Solution

As Vadim points out below I was using a old version of xgboost (the one offered on cran). Use the following to update the version and you will get 0.5 as expected.

install.packages("drat", repos="https://cran.rstudio.com")
drat:::addRepo("dmlc")
install.packages("xgboost", repos="http://dmlc.ml/drat/", type = "source")
$\endgroup$
5
  • $\begingroup$ Are you sure they aren't coded as -1, 1? $\endgroup$
    – Adrian
    Commented Feb 2, 2016 at 10:09
  • 1
    $\begingroup$ perhaps you have a standardization somewhere $\endgroup$
    – rapaio
    Commented Feb 2, 2016 at 11:11
  • $\begingroup$ see edits above @rapaio $\endgroup$
    – sachinruk
    Commented Feb 2, 2016 at 23:03
  • 1
    $\begingroup$ You really can't be sure that xgboost is not internally standardizing the variables without scouring the documentation and inspecting the source code, regardless of what your data looks like in R. That said, I don't see any reason for a tree based model to standardize. $\endgroup$ Commented Feb 2, 2016 at 23:07
  • $\begingroup$ This certainly seems to be the case as it happens on the basic example when I inspect the tree. $\endgroup$
    – sachinruk
    Commented Feb 3, 2016 at 3:25

1 Answer 1

1
$\begingroup$

This is most likely because your data is in sparse format. E.g., enforcing the data to be dense in the basic example (using as.matrix(train$data)) would show the splits to be at 0.5.

For sparse "dummy" data, one is the only actual value that a feature might have. Thus, the search for best split_value for such a feature doesn't really move far from its initial value of zero, mostly just the offset with rt_eps.

$\endgroup$
5
  • $\begingroup$ I'm afraid I've already done as.matrix to change the data frame. The data isn't sparse $\endgroup$
    – sachinruk
    Commented Feb 3, 2016 at 6:27
  • $\begingroup$ Then please provide a reproducible example. $\endgroup$ Commented Feb 3, 2016 at 7:02
  • $\begingroup$ As requested minimal example included $\endgroup$
    – sachinruk
    Commented Feb 3, 2016 at 11:27
  • 1
    $\begingroup$ This is the "dense matrix" from the "basic example". And I get the [1] "0.5" "0.5" "0.5" NA NA NA result from it using either the most recent or the pre-"brick" version of xgboost code on 64bit windows and linux. What xgboost code and what system are you running? $\endgroup$ Commented Feb 3, 2016 at 16:44
  • $\begingroup$ So it turns out that I had to update the xgboost via devtools and NOT the cran version. $\endgroup$
    – sachinruk
    Commented Feb 4, 2016 at 0:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.