8
$\begingroup$

I have a dataset with ~30k samples and 35 features (after feature selection; these seem to be the most important features for this dataset and they have low correlation between each other).

After doing grid search with 10-fold CV on the hyperparameters, to my surprise I get the lowest validation error when colsample_bytree is such that only 1 feature is sampled for each tree! (Edit: actually, with 2 features sampled per tree it works slightly better - but if I increase the number of features sampled per tree the performance keeps getting worse). The depth of each tree is 3 and I am building 2000 trees. That is, for each tree, a feature is randomly selected, and then xgboost tries to fit to residuals using only that feature.

That seems to be very unusual. How should I interpret this? If I have feature interactions in my trees, I start to overfit? But then I would expect performance with trees of depth 1 and no feature subsampling to perform just as good, yet they don't. In fact, in the grid search, nearly all models with such extreme feature subsampling did better than models without feature subsampling.

Edit: is it possible that I have some features that fit well to the training set but generalize very poorly, and such individual feature sampling helps to avoid those features dominating the model? I am struggling to see what else this could mean.

Edit2: Tried removing individual features, performance does not improve, which suggests that my hypothesis from the previous edit is unlikely. On the other hand, I found that the optimal performance is actually when I sample 2 features per tree. At least now my features are interacting, but still, I am not sure how to explain this gain in performance.

Edit3: This paper is somewhat relevant, but not really. In random forests, the optimal setting for feature subsampling is usually the square root of the number of features: "Influence of Hyperparameters on Random Forest Accuracy" by S. Bernard et. al., 2009. It is easier to see why it would be more useful in random forests, however, which rely on variability in the trees and do not fit to the residuals like XGBoost.

Improve this question
$\endgroup$
5
  • $\begingroup$ How did you do the feature selection? How did you do the hyperparameter grid-search? Your configuration looks very unlikely to me, did you try something like 6 depth, 0.8 subsample and then increase the number of trees (starting from 50 or so)? 2000 trees seem to be a lot! And 3 a small depth if the feature really are not correlated. $\endgroup$
    – Mayou36
    Commented May 31, 2017 at 15:41
  • $\begingroup$ For hyperparameter search, I tried all combinations of {100, 500, 1000, 2000} trees, {3, 5, 7} depths, {1, 0.9, 0.5, 0.1} subsampling, {1, 0.9, 0.5, 0.05} feature subsampling, {0.1, 0.01, 0.001} learning rate - all using 10-fold cross-validation (without reshuffling the folds). Then, I tried to narrow the optimal values down further around the best performing values. Latest results are 900 trees, 3 depth, 1 subsampling, min child weight 500, 0.07 feature subsampling (that is, 2 features out of 35 per tree!), 0.012 learning rate. $\endgroup$
    – rinspy
    Commented Jun 1, 2017 at 9:01
  • $\begingroup$ For feature selection I trained very simple xgboost models on all features (10 trees, depth 3, no subsampling, 0.1 learning rate) on 10-folds of cross-validation, selected the feature that had the greatest importance on average across the folds, noted that feature down and removed that feature and all features highly correlated with it from the set. I repeated that process until I had selected 35 features out of ~500. I'm getting the best validation and test set results with this approach so far... $\endgroup$
    – rinspy
    Commented Jun 1, 2017 at 9:05
  • $\begingroup$ I also think due to the nature of the data, there may be a lot of spurious correlations between the features and the response. The samples come from ~70 time series; those come from ~15 categories, and time series within each category are probably quite strongly correlated. I make sure that when I do cross-validation I test on samples that come from different time series than the ones seen during training. $\endgroup$
    – rinspy
    Commented Jun 1, 2017 at 9:06
  • 1
    $\begingroup$ You really should not grid search the number of trees (it's computationally very wasteful) or learning rate (smaller is better as long as you fit enough trees to reach an optimal hold out error). $\endgroup$
    – Matthew Drury
    Commented Sep 12, 2017 at 17:59

1 Answer 1

Reset to default
2
$\begingroup$

You seem to fine-tune the wrong things.

On your feature selection: I don't think that this is done properly:

  • You remove the good feature and all linearly correlated features. That's nice, but higher order correlated features are still there. On the other hand, strong correlation does not always mean that the feature is useless.
  • So you should keep the good feature in the set and remove all the features that are useless. The goal is to still have a high score in the end. This way you make sure you don't remove good features as you would notice it because the score decreases.
  • You should train a good model (at least once in a while) in order to know which features are helpful.

For the hyperparameter optimization:

  • you should fix some variables in the beginning (all except n_estimators), optimize (roughly) that parameter with a more fine grained grid (from 10 to 500 in steps of 20 for example).

  • My general suspicion: Way to many estimators, to low learning_rate (at this stage) and to shallow (set depth to 6). Try maybe the following:

    • eta = 0.2
    • n_estimators = [50...400]
    • subsample = [0.8]
    • depth = 6

    and leave the rest as is. Of course, those depend strongly on the data.

A nice guide for XGBoost hyperparameter optimization can be found here.

So I'd propose you to redo the feature selection keeping the good features in the set and sometimes use a good XGBoost configuration by optimizing it. Do not forget to maybe create a small holdout set which you do not use in the feature selection. This can be used in the end to know the real performance.

Improve this answer
$\endgroup$
7
  • $\begingroup$ On feature selection: I do not think there is any one way to do "proper" feature selection, short of evaluating all possible feature combinations. I know what the features mean in the domain and the ones I ended up with look reasonable. $\endgroup$
    – rinspy
    Commented Jun 1, 2017 at 12:24
  • $\begingroup$ In terms of hyper-parameter selection - I have searched the hyper-parameter space rather exhaustively. In any case, to make things concrete: depth=6, n_estimators=50, subsample=0.8, colsample_bytree=1, learning_rate=0.2, min_child_weight=1 results in avg. AUC 0.6738, 61.25% precision and 67.49% recall. Increasing the number of estimators keeps it in the same general range (e.g. with 250 estimators AUC is 0.6724, 60.84% precision, 70.09% recall). $\endgroup$
    – rinspy
    Commented Jun 1, 2017 at 12:30
  • $\begingroup$ Well, not all, but if you train on all features and remove some which do not decrease the score, you more or less could get rid of all those. This would reduce the amount most probably quite good. Or try it on your features: remove one feature and see whether the score decreases. $\endgroup$
    – Mayou36
    Commented Jun 1, 2017 at 12:31
  • $\begingroup$ Compare with my best results so far which are depth=3, n_estimators=900, subsample=1, colsample_bytree=0.07, learning_rate=0.012, min_child_weight=500 that give avg. AUC 0.7718, 75.05% precision, 63.86% recall - a very big improvement which I can only get when I set colsample_bytree to a very low value (such that very few features are sampled per tree!) Having a high number of trees then ensures that all (or nearly all) of the possible 2-feature-combinations are tried in at least one tree. $\endgroup$
    – rinspy
    Commented Jun 1, 2017 at 12:32
  • $\begingroup$ I have tried that - removing each feature makes the performance slightly worse. Anyway, I think that is beside the point - we can just assume the 35 features are all I have to work with. $\endgroup$
    – rinspy
    Commented Jun 1, 2017 at 12:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.