I'm currently testing some feature selection methodologies/algorithms in R, like the Recursive Feature Elimination from the R caret package, and also the RRF R package, to select a subset of features from a merged dataset, which is comprised by both gene expression data, and some continuous clinical variables. My main goal, is to inspect via any of these methodologies, in any(or most) of the clinical variables are selected along with the gene features, in the "subset" returned from the feature selection procedure, in order then to test this "composite" set in downstream procedures. My main question(although might sound very naive), is about the random seed issue and the reproducibility of my results. In detail, both the rfe function from caret, as also the RRF function from the RRF package, need before each run, a random seed number in order for the results to be reproducible. However, how i can ensure via a more "appropriate" methodology, that the subset of features selected is similar, and not just dependent on the specific random seed ?

In other words, should i test with either methodology, a set of different random seed numbers, through the argument:


and then inspect which features are commonly selected ? And implement this somehow with a for loop ?

I insist on this matter, because i would like to perform various statistical methodologies with this selected subset, and it would be preferable to be as much as less "random".

  • 1
    $\begingroup$ To repeat a non-deterministic procedure 10 or 20 times to evaluate the variability is always recommendable. If not implemented in caret, try wrap your current procedure in a lapply function iterating with a random seed and collect all results in a list. Apply some metric evaluating how much in agreement the individual results are. Then, decide if one single run is stable enough, or perhaps the average of multiple runs is stable enough. If not stable enough, then change procedure. $\endgroup$ – Soren Havelund Welling Feb 22 '16 at 12:15
  • $\begingroup$ Dear Soren, thank you for your answer !! So if i have understood well, i have to creat something like a "sequence" of different random seed numbers-iterations, and inspect if common features appear repeatedly, right ? And if my notion is correct, for instance your suggestions of i.e. 20 different random seed numbers, would be "enough" ? $\endgroup$ – Jason Feb 22 '16 at 12:21
  • $\begingroup$ Yes, if you would like to narrow down your pool of features to one selection of those consistently scoring high on variable importance. You may find some features are always included, some are sometimes and someone are never included. Thus a clear threshold is not always obvious. If 20 repeats of the procedure did provide a consistent answer, I might rather modify the procedure (more trees etc.) than increasing repeats. $\endgroup$ – Soren Havelund Welling Feb 22 '16 at 12:41
  • $\begingroup$ Actually, although im a "newbie", rfe from caret and guided random forest selection work differently--for instance regarding guided random forest, i used 1,000 as a number of trees, while the rfe has other parameters (but i think that i can also define the number of trees). Also, rfe also "trains" the data with the selected features, while with RRF, uses some "kind" of penalization --dl.dropboxusercontent.com/u/45301435/GRF.pdf --Perhaps, i could use a number of different random seeds for both methodologies, and then inspect common features both within and between the methods. $\endgroup$ – Jason Feb 22 '16 at 12:51
  • 1
    $\begingroup$ I have speed read the paper and package. My first impression(correct me if I'm wrong) is that RRF only focus on gini gain criterion. As far as I know Gini is not preferred over permutation variable importance exactly because Gini is notoriously biased and unstable. How well RRF fixes this, I don't know. See this paper for a gold-standard solution: www.ncbi.nlm.nih.gov/pubmed/17254353 $\endgroup$ – Soren Havelund Welling Feb 22 '16 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.