1
$\begingroup$

I am trying to predict sales of certain product using regression method. I am using XGboost and using MAPE as final metric for comparison between models. I have around 23 features but there are many categorical variables which i have converted into dummy variables. So now there around 210 features many of which are sparse.

I ran XGBoost model on this and i checked for feature importance using xgb.importance(). It showed the importance value for only 84 features. So i ran one more iteration of XGBoost only with these 84 features which are important but there is no change in model output.

So does presence of other features which is not important has any affect on XGBoost model ? How can i perform feature selection using XGBoost ?

Improve this question
$\endgroup$
1
  • $\begingroup$ I do not know about these techniques (XGboost or what the acronym MAPE stands for), but it seems like these already incorporate some sort of feature selection for the final model. That, or the other features have such little influence on the model estimates that the difference between in- or excluding them is not visible due to rounding. So, my suggestion would be to check the package information for any inherent selection procedures $\endgroup$
    – IWS
    Commented Feb 27, 2017 at 9:23

2 Answers 2

Reset to default
1
$\begingroup$

Your result is correct, XGB recognizes that many of your features are not important and didn't use them in the process of building decision trees. You can force XGB to use all of them by increasing max tree depth setting, but you are overfitting the data this way.

Back to your problem, only 84 features are used by XGB and therefore discarding others produces very similar result

Improve this answer
$\endgroup$
0
$\begingroup$

As iws said, xgboost does the feature selection for you. There are three features importance index: gain, cover and frequency. If the feature's gain is low, then it contribute little to the error reduction. So remove it will not affect the results.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Do you mean to say that we don't have to worry about variable selection in xgboost ? I was under the impression that variable selection has to be done irrespective of the algorithm. Can you provide any supporting documents for this ? $\endgroup$
    – navinkb
    Commented Mar 29, 2017 at 5:51
  • $\begingroup$ @navinkb stats.stackexchange.com/questions/1292/… $\endgroup$
    – wolfe
    Commented Mar 29, 2017 at 10:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.