1
$\begingroup$

Experimental results

Basic info abut the experiment:

  • Binary classification of exons
  • 10 fold cross validation
  • 1200 features of exons are ranked by Fisher Score, Relief and Gini Index feature selection algorithms
  • 1-nearest neighbor classifier is applied to the ranked list of features.
    • The classifier is trained on 1 best feature, on 2 best features, on 3 best features, ...

There is no significant difference between Random Feature Selection and Automatic Feature Selection.

John, Kohavi, and Pfleger defines the following categories of features: John, Kohavi, and Pfleger defines the following categories of features

What are some potential reasons for no significant difference between Random Feature Selection and Automatic Feature Selection? Does that mean that all of 1200 features are strongly relevant?

Reference: G. H. John, R. Kohavi, and P. Pfleger. Irrelevant features and the subset selection problem. In Machine Learning: Proceedings of the Eleventh Inter- national Conference. Morgan Kaufmann, 1994.

$\endgroup$
  • 2
    $\begingroup$ It appears that you are using improper accuracy scoring rules. See this. $\endgroup$ – Frank Harrell May 27 '18 at 11:44
  • $\begingroup$ I tried to use AUROC. AUROC is a continuous accuracy score. I conducted a new experiment with the help of 3NN classifier. I obtained approximately the same results. So there is no significant difference between Random Feature Selection and Automatic Feature Selection. Could you please confirm that AUROC is a proper accuracy scoring rule for my case? Thank you in advance for your help. $\endgroup$ – user162352 May 27 '18 at 15:25
  • 1
    $\begingroup$ AUROC is semi-proper. It does not give adequate weight to extreme predictions that are right. So it lacks statistical power. A proper score would be the Brier score or the logarithmic scoring rules (which is a function of the log-likelihood). Any function of the log-likelihood, including pseudo $R^2$ will provide maximum statistical utility. $\endgroup$ – Frank Harrell May 27 '18 at 17:05
  • $\begingroup$ Your feature selection might be too noisy? $\endgroup$ – kjetil b halvorsen May 27 '18 at 18:51
  • $\begingroup$ @FrankHarrell I calculated Brier score. Here is a result: prnt.sc/jnbf5p There is still no significant difference between Random Feature Selection and Automatic Feature Selection. Do you have an idea of some potential reasons for that? $\endgroup$ – user162352 May 27 '18 at 19:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.