I am new to ML, so please pardon my ignorance.

I have 12 features (left), and the column headers are different feature selection methods. Dark green indicates the a stronger predictor while white indicates a weaker predictor (I think for Fisher scores, a lower index corresponds to a better rank and thus a better predictor?)

My problem is that the 6 different methods are not as consistent as they should be. Is this expected, and if so, which features should be picked?

On a related note, these methods were performed on X_train and y_train. Should it be done on the X and y instead of the training sets?

Many thanks.

enter image description here

  • $\begingroup$ There is no guarantee that these methods will converge on the "true" model. Why do you expect these methods to be consistent? $\endgroup$
    – dimitriy
    Jan 12, 2023 at 1:43
  • $\begingroup$ Ahh, I see. I expected the methods to be consistent because their purpose was the same, but after seeing, for example, the BSA being predicted as a great predictor in the first 3 and poorly in the last 3, I thought that maybe there was an inconsistency involved in the methods. Also, was it ready to apply these methods on the test set, or should it be done on the entire dataset? $\endgroup$
    – djdumpling
    Jan 12, 2023 at 1:57
  • $\begingroup$ I think the train set is the standard approach. $\endgroup$
    – dimitriy
    Jan 12, 2023 at 2:02

1 Answer 1


You’re using different criteria to select the important features. The whole point of having different criteria is to be able to select features that are important according to what you value. If they gave the same answer, they would be boring.

Further, feature selection is notoriously unstable, as Frank Harrell has discussed many times; I like his presentation here that starts around the 15-minute mark. I suggest that you question why you should do feature selection at all.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.