My objective is to find genes that can be used as biomarkers with low error.

I am using Random Forest (RF) using R package randomForest and following the steps in below link as it is has similar objective but made some changes.


So I am interested in finding a gene signature differentiating two groups of samples. I ran random forest with >6000 differentially expressed genes (variables) with 10 different seeds, ntree=30,000 and and high mtry to find important genes. OOB error was 11.81%. Then I selected the 100 most important genes from each RF (meandecreaseaccuray, as it was more correlated to -logpvalue). Then I found most common genes among them, 81.

Then I made new RF using 81 genes, ntree=30,000, default mtry and recalculated importancemeandecreaseaccuracy.

Now I started with combining top important genes until the OOB error is reduced to minimum and found top 4 genes reduced the OOBerror to 5.6%, however I used 50,000 tree and mtry=1. The reason for mtry=1 is as the tutorial suggested these genes are corregulated. Also as per an explanation here, I want each selected variable to have equal importance for classification, hence mtry=1 should be appropriate.

I talked to someone who is computational biologist, unlike me who is molecular biologist, and he said my RF is over-fitting and I need to cross valdiate, without giving any suggestion.

Based on my reading of posts on stat exchange and others, I find the following: 1) RF seldom over-fits for classification problem 2) RF doesn't need cross validation as OOBerror is doing the same thing.

My question is this: Is my approaching of filtering important genes and running new RF using lesser genes to find best signature conceptually wrong? So if my model if overfitting for firt 4 variable (OOBerro =5.6%), shall I consider all the variable until my OOBerror is same as with all the variables? Would that be the correct gene signature.

Also, I am planning to do ROC analysis to see if my gene signature is statistically sound (which I think it should be as they are deferentially expressed genes with very low fdr and good fold change and I have plotted box plot to see the separation of data). Is this approach redundant?

I will appreciate any help as I just want to learn more about this exciting tool.

P.S.: I had an older post that I re-framed with better question title.


1 Answer 1


You are overfitting because you are running multiple models and testing them always on the same data set. This will lead to overfitting and an apparent increase in accuracy.

The general procedure in ML is training -> validation -> testing. For each step you need a different and independent data set. Sometimes though, you cannot afford to split your data into so many partitions, for ex. genome data, so you use the one and only data set for all three steps. This is not great but often times is the only option. However, in this case you have to be extra careful or you will very quickly overfit your model/s.

Basically, you would build many models on your data and then test them once and only once using CV / OOB. If you use the results obtained from a CV / OOB to build an additional model you are overfitting, since you are using results obtained from your data. To build an additional model you would need an additional data set to test the second model.

  • $\begingroup$ Thanks for explaining that. But I wish to reduce the number of variable that will give me same prediction as all variables. So I used the top ranked features. Could you enlighten me if use X number of variable that doesn't overfit (has same OOB error as the all variables) and use the X as signature? Or that approach is faulty. I don't have additional data to test unless I divide my data into two parts, but then the number of data to generate the predictors wouldn't be good. $\endgroup$ Commented Feb 12, 2019 at 1:20
  • $\begingroup$ @PiyushJoshi I understand, data is hard to come by, especially for genomic data. But rules must be followed, otherwise you will very quickly overfit. Also forgot to mention, OOB si very similar to CV, so you do not need CV with RF. As for selecting the top X variables... you shouldn't do it, since you would be testing your new model with top X variables on the same data set, which will result in optimistic results. What you could do is build your model once, select top X variables, build second model and keep it. No further processing, but keep in mind that this is double dipping. $\endgroup$ Commented Feb 12, 2019 at 7:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.