1
$\begingroup$

I have a dataset of 100 patients and 1500 features. I split 80 train 20 test first and then use the train set to get the best hyperparameters / best feature subset doing the following:

I randomly split the train set into 70% train 30% "test" (I don't use it) X times and pass it through a classifier (Random Forest, L1 Logistic Reg, etc). Then, I take the most important features (from feature importance in RF or weights in LR) passing a certain threshold (mean/median of weights/importance). For each split, I save this list and make a counter. After X iterations, I have a counter like {feature 1: Y times, feature 2: Z times, etc}. Then I perform backwards elimination on the top features using the complete train set (the 80% of the complete dataset) and use K-fold stratified CV to find the hyperparameters for each BE iteration.

My mainly concern lies in the feature counter I am creating. I am using 70% of 80% of the total dataset to get these features. Note that I never use the 20% test set I set aside at the very beginning. Am I overfitting at any step?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Step 1: Split the data base in 80/20 (you do not use 20 for anything now)

Step 2: The 80 of step 1, you split in 70/30.

Step 3: You use the 70/30 of step 2 to find the most importat features.

Step 4: You do k-fold cv using 80/20 to choose the best model.

I think You are doing great in terms of spliting the database.

You may also consider to choose your features using a procedure like this:

1) Permutation importance: The permutation feature importance is defined to be the decrease in a model score when a single feature value is randomly shuffled

2) Shap values: It is not an easy concept since it is based in game theory, but it shows the importance of each feature.

Improve this answer
$\endgroup$
4
  • $\begingroup$ In step 3, I am splitting the train test into 70-30 X times. That means that at the end I do use all the train set to find the subset of features. In step 4, I do k-fold cv on the 80% train (i.e. all data points from step 3) $\endgroup$
    – Luis Pinto
    Commented Feb 26, 2020 at 23:10
  • $\begingroup$ I cannot see any problem in your choice. It is not just very clear how you choose your best features. $\endgroup$
    – DanielTheRocketMan
    Commented Feb 26, 2020 at 23:15
  • $\begingroup$ For every 70% of the train set, I pass it through a classifier and take the features passing a feature threshold (I use weights or feature importance) and create a counter out of these lists. I use "SelectFromModel" function from sklearn.feature_selection package. $\endgroup$
    – Luis Pinto
    Commented Feb 26, 2020 at 23:20
  • 1
    $\begingroup$ It is ok. An option to do that ... See the edition. Just a minute. $\endgroup$
    – DanielTheRocketMan
    Commented Feb 26, 2020 at 23:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.