# XGBoost Feature Importance Changes with Random Seed

1. Analysis Goal: Identify features that provide an accurate prediction of a binary outcome and also explain how the features are related to the output
2. Data: 72 features and 200 instances.
3. Process: Sampled 70% of the data with a random seed and trained an XGBoost model. With different random seeds, variable importance and prediction performances change.
4. Some more detail: Using the fitted model on the training data, I have calculated Shapley values for all training instances for the top 4 important features. For each of these features, I am plotting the feature values versus Shapley values to identify some thresholds that can result in a positive or negative log(odds) of the event of interest. I have also tested recursive partitioning. The thresholds keep changing with different seed values and AUC(ROC) is almost 20% lower than XGBoost. Finally, there is nothing holy about 4 features (as opposed to 5 or 6 or all), I have chosen them for the sake of simplicity in the interpretations.
5. Problem: The thresholds and top 4 features keep changing whenever I change the random seed. Is there a systematic way to make some inferences like the ones I am looking for with confidence?

I have looked at other posts here but could not really find answers (e.g. this, this, this).

• I'd not be surprised changing results with 200 instances. Why don't you repeat your experiment multiple times and report top-top features, i.e. feature X appeared 85 times in top-4, feature Y 72 times etc. Mar 26, 2022 at 17:39
• I have tried something like this and chose a seed with the most common feature importance occurrence. I think this solves it. Thank you. Mar 26, 2022 at 20:00
• Be careful using Shapley values. This seems to not be the approach you want. If you want to get the best possible model, try conditional feature importance methods with recursive feature selection/elimination. if you want to understand the relationship between each variable and the response, the marginal feature importance may be more ideal for you. May 7, 2022 at 17:10
• Thank you for your response, Joe. Upon a quick google search, I found your paper on UMFI. I will read through it, compare it with the Shapley approach, and be back with questions. May 8, 2022 at 18:08