1
$\begingroup$
  1. Analysis Goal: Identify features that provide an accurate prediction of a binary outcome and also explain how the features are related to the output
  2. Data: 72 features and 200 instances.
  3. Process: Sampled 70% of the data with a random seed and trained an XGBoost model. With different random seeds, variable importance and prediction performances change.
  4. Some more detail: Using the fitted model on the training data, I have calculated Shapley values for all training instances for the top 4 important features. For each of these features, I am plotting the feature values versus Shapley values to identify some thresholds that can result in a positive or negative log(odds) of the event of interest. I have also tested recursive partitioning. The thresholds keep changing with different seed values and AUC(ROC) is almost 20% lower than XGBoost. Finally, there is nothing holy about 4 features (as opposed to 5 or 6 or all), I have chosen them for the sake of simplicity in the interpretations.
  5. Problem: The thresholds and top 4 features keep changing whenever I change the random seed. Is there a systematic way to make some inferences like the ones I am looking for with confidence?

I have looked at other posts here but could not really find answers (e.g. this, this, this).

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ I'd not be surprised changing results with 200 instances. Why don't you repeat your experiment multiple times and report top-top features, i.e. feature X appeared 85 times in top-4, feature Y 72 times etc. $\endgroup$
    – gunes
    Commented Mar 26, 2022 at 17:39
  • 1
    $\begingroup$ I have tried something like this and chose a seed with the most common feature importance occurrence. I think this solves it. Thank you. $\endgroup$
    – Foolish Frog
    Commented Mar 26, 2022 at 20:00
  • 1
    $\begingroup$ Be careful using Shapley values. This seems to not be the approach you want. If you want to get the best possible model, try conditional feature importance methods with recursive feature selection/elimination. if you want to understand the relationship between each variable and the response, the marginal feature importance may be more ideal for you. $\endgroup$
    – Joe Janssen
    Commented May 7, 2022 at 17:10
  • $\begingroup$ Thank you for your response, Joe. Upon a quick google search, I found your paper on UMFI. I will read through it, compare it with the Shapley approach, and be back with questions. $\endgroup$
    – Foolish Frog
    Commented May 8, 2022 at 18:08

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.