I wonder if by implementing a feature selection technic using training with a specific algorithm you can select the feature you need to use with other algorithms also.
To be more specific after I trained an XGBoost model with the default parameters and using the following code:

columns = ['Thresh', 'n', 'selected_features', 'Logloss']
feature_selection = pd.DataFrame(columns=columns)
thresholds = np.sort(model.feature_importances_)
for thresh in thresholds:
    # select features using threshold
    selection = SelectFromModel(model, threshold=thresh, prefit=True)
    select_X_train = selection.transform(X_train)
    # train model
    selection_model = XGBClassifier(objective='multi:softprob', n_jobs=-1)
    selection_model.fit(select_X_train, y_train)
    # eval model
    select_X_val = selection.transform(X_val)
    y_pred = selection_model.predict_proba(select_X_val)
    score = log_loss(y_val, y_pred)
    result = pd.DataFrame(
            thresh, select_X_train.shape[1],
            X_train.columns[selection.get_support()], score
        ]).reshape(1, -1),
    feature_selection = feature_selection.append(result, ignore_index=True)

I ended up with:

enter image description here

and I decided to continue with the 22 out of 33 features. My question is can I use these features as the most important to try other models (including DNN), or their importance is specific for this algorithm?

  • $\begingroup$ Good question. (+1) Your intuition that this is specific to the algorithm used is correct. Please see my detailed answer below. $\endgroup$
    – usεr11852
    Dec 15, 2018 at 12:27

1 Answer 1


Feature importance measurements are strongly associated with the underlying model. That is because different models can utilise a feature differently. Some (like a gradient boosting machine (GBM)) will recursively partition it while others (like a generalised additive model (GAM)) might fit a spline on it. In addition, the way that an algorithm accounts for interactions between variables might be completely different; for example, in a random forest (RF) interactions are essentially detected in an automated way as deeper trees are grown by sequentially examining different features, while in standard linear model (LM) we would need to explicitly define it.

That said, all is not lost, the feature importance from an GBM algorithm does hold some relevance for a GAM model or a neural network (NN) as well. It is totally unreasonable to expect that a given algorithm most important features are useless for the learning task of another algorithm. They will contain relevant information but there is little guarantee that these features will present the optimal set of features for another type of learner too. We can definitely use them as a first educated guess.

A final note: especially for a deep neural network (DNNs) where we commonly have hundreds (if not thousands) of input features it is not always coherent to examine individual variable importance. DNNs are strong exactly because they can synthesize features internally and train themselves on those internal features. For instance, within a image classifier the value of a single pixel is often immaterial, it is the convolution layers that make the big difference. Overall, we might wish to exclude a particular feature (or better yet replacing it with pure noise), retrain our DNN on this new dataset and then compare the performance difference but this method can turn out to be very computationally expensive for anything more than simple cases. There are some further techniques for DNNs that give feature importance for a DNN (e.g. Sundararajan et al. "Axiomatic Attribution for Deep Networks" (ICML 2017)) but once again those are particular to the learner used.

  • $\begingroup$ Welcome to the CV community Yanni! $\endgroup$
    – usεr11852
    Dec 15, 2018 at 12:24
  • $\begingroup$ Thank you both for your thorough answer and your welcome! Cheers. $\endgroup$
    – Yannis
    Dec 15, 2018 at 14:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.