Feature Importance of a feature in lightgbm is high but reduces evaluation score I am currently working on a machine learning project using lightGBM.
When I added a feature to my training data, the feature importance result I got from lgb.plot_importance(gbm, max_num_features=10)is high, but adding this feature reduced the RUC_AUC_score for performance evaluation.
In pursuing high prediction, do we just drop this feature?
Or theoretically there are few concerns I need to investigate about before dropping it?
My current understanding is that the feature importance in light gbm shows how many splits occurred on the feature when the data is trained, so it does not necessary mean that high feature importance always leads to better performance. So in the case above I can just drop the feature according to the decrease in evaluation score.
I have been pondering about this question for few days and still can come up to a solid conclusion by myself.
If any of my thought is wrong please kindly let me know.
Thanks. 
 A: If you look in the lightgbm docs for feature_importance function, you will see that it has a parameter importance_type. The two valid values for this parameters are split(default one) and gain. It is not necessarily important that both split and gain produce same feature importances. There is a new library for feature importance shap.
Go through this article written by the co-author of above library, he explains this is much better way.
A: You should use verbose_eval and early_stopping_rounds to track the actual performance of the model upon training. For example, verbose_eval = 10 will print out the performance of the model at every 10 iterations. It is both possible that the feature harms your model or the model is overfitted. In overfitting situations usually there occurs a dominating feature as like your case.
By tracking the performance visually upon training by verbose_eval and ensuring that the model is not overfitting by early_stopping_rounds, you can analyze the features by taking them one by one off to see how the performance is affected directly. Do not take off multiple features at once, the model can lose a strong pattern from those and you can just misinterpret the result. 
