Random Forest feature selection over-fitting doubt

Question

My objective is to find genes that can be used as biomarkers with low error.

I am using Random Forest (RF) using R package randomForest and following the steps in below link as it is has similar objective but made some changes.

https://horvath.genetics.ucla.edu/html/RandomForestScreening/SupplementRandomForests.pdf

So I am interested in finding a gene signature differentiating two groups of samples. I ran random forest with >6000 differentially expressed genes (variables) with 10 different seeds, ntree=30,000 and and high mtry to find important genes. OOB error was 11.81%. Then I selected the 100 most important genes from each RF (meandecreaseaccuray, as it was more correlated to -logpvalue). Then I found most common genes among them, 81.

Then I made new RF using 81 genes, ntree=30,000, default mtry and recalculated importancemeandecreaseaccuracy.

Now I started with combining top important genes until the OOB error is reduced to minimum and found top 4 genes reduced the OOBerror to 5.6%, however I used 50,000 tree and mtry=1. The reason for mtry=1 is as the tutorial suggested these genes are corregulated. Also as per an explanation here, I want each selected variable to have equal importance for classification, hence mtry=1 should be appropriate.

I talked to someone who is computational biologist, unlike me who is molecular biologist, and he said my RF is over-fitting and I need to cross valdiate, without giving any suggestion.

Based on my reading of posts on stat exchange and others, I find the following: 1) RF seldom over-fits for classification problem 2) RF doesn't need cross validation as OOBerror is doing the same thing.

My question is this: Is my approaching of filtering important genes and running new RF using lesser genes to find best signature conceptually wrong? So if my model if overfitting for firt 4 variable (OOBerro =5.6%), shall I consider all the variable until my OOBerror is same as with all the variables? Would that be the correct gene signature.

Also, I am planning to do ROC analysis to see if my gene signature is statistically sound (which I think it should be as they are deferentially expressed genes with very low fdr and good fold change and I have plotted box plot to see the separation of data). Is this approach redundant?

I will appreciate any help as I just want to learn more about this exciting tool.

P.S.: I had an older post that I re-framed with better question title.

Answer 1

You are overfitting because you are running multiple models and testing them always on the same data set. This will lead to overfitting and an apparent increase in accuracy.

The general procedure in ML is training -> validation -> testing. For each step you need a different and independent data set. Sometimes though, you cannot afford to split your data into so many partitions, for ex. genome data, so you use the one and only data set for all three steps. This is not great but often times is the only option. However, in this case you have to be extra careful or you will very quickly overfit your model/s.

Basically, you would build many models on your data and then test them once and only once using CV / OOB. If you use the results obtained from a CV / OOB to build an additional model you are overfitting, since you are using results obtained from your data. To build an additional model you would need an additional data set to test the second model.