1
$\begingroup$

I applied several ML algorithms to my data. My data is made of several predictive numeric features and a target categorical binary feature.

My aim is to build a classifier and predict the target class, based on those predictive features (the target variable is binary). After running multiple algorithms like XGBoost and Lasso regression for example, and after checking feature importance, I get totally different feature importance for each algorithm. What is the meaning of this? look at Fibroblasts for example, it has 100% importance in one algorithm and zero in the other.

Feature importance of XGBoost model:

enter image description here

Feature importance of Lasso Regression model:

enter image description here

Improve this question
$\endgroup$
3
  • $\begingroup$ Identifying important features from data is often futile, especially when we are not talking about big data. How many observations are there and is there multicollinearity? Is fibroblasts correlated with any feature of importance in LASSO? $\endgroup$
    – Bernhard
    Commented Aug 17, 2022 at 10:31
  • $\begingroup$ To the general problem I would like to hint you to a talk by Prof. Frank Harrell on YouTube youtu.be/DF1WsYZ94Es?t=1011 where at minute 18:00 he says "the simplest way to say it: the probability to select the "right" variables is zero." I know this goes against what is propagated but maybe have a look at the talk and see if what he says suits your situation. $\endgroup$
    – Bernhard
    Commented Aug 17, 2022 at 10:38
  • $\begingroup$ Feature importance is a property of both the data and the model together, so it's not surprising that two different models would compute two different importances. Moreover, even if you have the same data and the same model, two different importance metrics can produce different rankings of the features. $\endgroup$
    – Sycorax ♦
    Commented Dec 29, 2022 at 23:19

2 Answers 2

Reset to default
1
$\begingroup$

The short answer is that you likely have multiple variables in the data that measure similar things - so one model picks up one way of measuring that idea (Fibroblasts) and the other picks up a different way (some combination of other variables that are correlated with fibroblasts).

Check for correlations between your predictor variables and consider removing some. I don't know how big your dataset is, but I'm guessing you have too many candidate predictor variables anyway - this type of swing is usually seen in small datasets. There's a "rule of thumb" floating around that you should have 10 events per variable at minimum, but the justification for that rule is weak at best: https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-016-0267-3.

Improve this answer
$\endgroup$
1
$\begingroup$

Adding to above answer:

In addition to possibly having multiple features having redundant information (different algorithm picking one redundant feature over the other), it is worth noting that feature importance means differently between a regression model vs a tree based model and this may cause preferential treatments on features.

Tree based models uses splits to make decisions and its feature importance is usually a measurement of the reduction in entropy when performing the splits. This means it likes features that has a lot of variation (i.e.: a lot of distinctive values / continuous features).

Regression is an additive model. Its feature importance is usually the coefficient of the variable. This would mean a feature importance score from regression would favor variables where a small change would result in big swings in prediction.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.