# What are some potential reasons for no significant difference between Random Feature Selection and Automatic Feature Selection?

Basic info abut the experiment:

• Binary classification of exons
• 10 fold cross validation
• 1200 features of exons are ranked by Fisher Score, Relief and Gini Index feature selection algorithms
• 1-nearest neighbor classifier is applied to the ranked list of features.
• The classifier is trained on 1 best feature, on 2 best features, on 3 best features, ...

There is no significant difference between Random Feature Selection and Automatic Feature Selection.

John, Kohavi, and Pfleger defines the following categories of features:

What are some potential reasons for no significant difference between Random Feature Selection and Automatic Feature Selection? Does that mean that all of 1200 features are strongly relevant?

Reference: G. H. John, R. Kohavi, and P. Pfleger. Irrelevant features and the subset selection problem. In Machine Learning: Proceedings of the Eleventh Inter- national Conference. Morgan Kaufmann, 1994.

• It appears that you are using improper accuracy scoring rules. See this. – Frank Harrell May 27 '18 at 11:44
• I tried to use AUROC. AUROC is a continuous accuracy score. I conducted a new experiment with the help of 3NN classifier. I obtained approximately the same results. So there is no significant difference between Random Feature Selection and Automatic Feature Selection. Could you please confirm that AUROC is a proper accuracy scoring rule for my case? Thank you in advance for your help. – user162352 May 27 '18 at 15:25
• AUROC is semi-proper. It does not give adequate weight to extreme predictions that are right. So it lacks statistical power. A proper score would be the Brier score or the logarithmic scoring rules (which is a function of the log-likelihood). Any function of the log-likelihood, including pseudo $R^2$ will provide maximum statistical utility. – Frank Harrell May 27 '18 at 17:05
• Your feature selection might be too noisy? – kjetil b halvorsen May 27 '18 at 18:51
• @FrankHarrell I calculated Brier score. Here is a result: prnt.sc/jnbf5p There is still no significant difference between Random Feature Selection and Automatic Feature Selection. Do you have an idea of some potential reasons for that? – user162352 May 27 '18 at 19:53