My objective is to find genes that can be used as biomarkers with low error.
I am using Random Forest (RF) using R package randomForest and following the steps in below link as it is has similar objective but made some changes.
So I am interested in finding a gene signature differentiating two groups of samples. I ran random forest with >6000 differentially expressed genes (variables) with 10 different seeds, ntree=30,000 and and high mtry to find important genes. OOB error was 11.81%. Then I selected the 100 most important genes from each RF (meandecreaseaccuray, as it was more correlated to -logpvalue). Then I found most common genes among them, 81.
Then I made new RF using 81 genes, ntree=30,000, default mtry and recalculated importancemeandecreaseaccuracy.
Now I started with combining top important genes until the OOB error is reduced to minimum and found top 4 genes reduced the OOBerror to 5.6%, however I used 50,000 tree and mtry=1. The reason for mtry=1 is as the tutorial suggested these genes are corregulated. Also as per an explanation here, I want each selected variable to have equal importance for classification, hence mtry=1 should be appropriate.
I talked to someone who is computational biologist, unlike me who is molecular biologist, and he said my RF is over-fitting and I need to cross valdiate, without giving any suggestion.
Based on my reading of posts on stat exchange and others, I find the following: 1) RF seldom over-fits for classification problem 2) RF doesn't need cross validation as OOBerror is doing the same thing.
My question is this: Is my approaching of filtering important genes and running new RF using lesser genes to find best signature conceptually wrong? So if my model if overfitting for firt 4 variable (OOBerro =5.6%), shall I consider all the variable until my OOBerror is same as with all the variables? Would that be the correct gene signature.
Also, I am planning to do ROC analysis to see if my gene signature is statistically sound (which I think it should be as they are deferentially expressed genes with very low fdr and good fold change and I have plotted box plot to see the separation of data). Is this approach redundant?
I will appreciate any help as I just want to learn more about this exciting tool.
P.S.: I had an older post that I re-framed with better question title.