I'm currently playing with caret R package, with one merged microarray affymetrix dataset(paired tissue samples), in order to build and test various classifiers, mostly based on trees-such as random forests and boosted trees to predict a binary outcome-categorical, that is disease status (normal and cancer status). In detail, my main "introductory" questions are the following:

1) Because the sample size of my dataset is somehow described as "medium"-60 samples(maybe even small), should i split my dataset like 60% training and 40% test set ? Or is more appropriate especially for random forests, because of the out of bag error-OBB( because is like an independent "testing set") to use all my dataset, and test independently in other dataset(s) with similar design/status ?

2) Moreover, except my genes-predictors, i also have 8 continuous PET variables, which also describe the same patients from my specific dataset. Thus, my main interest, is if the addition of these PET variables improve the outcome/prediction. An illustrative example of my PET variables is as follows (samples in the rows, PET variables in the columns):

enter image description here

Thus, after I construct a total data.frame of both genes and these PET variables in the columns, should i also transform-"preprocess" my total variables, like centering or scaling ? Or for the mentioned classification algorithms it does not matter, because RF can handle the "different nature" of the variables ? and the same applies with gbm method ? And overall is it correct to use different variables with RFs ?

3) Moreover, in case I want to build a ROC curve, to see the performance of the classifier based on the training set, how could I use similarly the pROC package from the below caret tutorial ?


In detail, from the above tutorial, the object gbmtune is the output of the train function. How could I use it similarly with the roc function, in order to create the ROC curve?

4) Finally (one more general question not explicitly related to my specific dataset), because in one other dataset I have a great number of predictors, is it appropriate to use first a correlation filter to remove redundant features, and then continue with the RFE methodology to "identify" the "best performers?

Please excuse me for any naive questions, but there are fundamental in order to have an appropriate approach on this task!


1 Answer 1


1: Yes you should either use OOB-CV with 5000 trees and/or 10-fold 20-repeated. Training on only 60% is likely to yield a pessimistic cross validation. With such few samples, any single CV will vary considerably. Repeat the 10-fold CV many times and compute the average score(+/- SD). Do not pool predictions from repeated CV and then after compute the statistics. That will yield a over optimistic CV. If using OOB-CV only, then train 5000-10000 trees to achieve a stable result. After CV, use all samples to train a single model.

2: Any decision tree model, such as random forest and gbm (gradient boosting machine, uses trees as default) handle features as non-parametric and is invariant to any monotonic transformation. Thus, no scaling etc. is needed. It's a fine start just to mix all features from both sources. You may want to perform some modest variable selection on your features, as most out of 5000 micro-array features are expected to be less useful. Don't search for a magic perfect combination of ~7 variables, but narrowing the field of potential useful variables may yield a modest improvement. Try e.g. to pretrain a RF and drop ~50-90% of variables with lowest permutation variable importance (never use gini). Variable selection needs to be cross-validated also. The direct OOB-CV error after variable selection is over optimistic. I was in similar situation training a RF model with 41 samples(very few) and 250 features(reduced to 30 features). Variable selection seemed to yield a modest prediction accuracy improvement in outer cross-validation and in external test sets.

3: For any classifier which yields not only labeling, but also yields some probability estimate, can be used to form a ROC plot. A ROC-plot communicates any attainable combination of selectivity and sensitivity. Here is ROC example for randomForest. R package for Weighted Random Forest? classwt option? . In the slides [slide 16], the caret package is used to tune a gbm model with a ROC metric in a inner calibration loop, [edited] but not cross-validated with a outer validation loop. Thus, the tuning only selects a good set of hyper-parameters but do not provide an unbiased CV estimate. For that, wrap the tuning in a outer cross-validation manually The tuning/calibration metric does not have to match the cross-validation metric. To form a ROC plot you need the true labels(binary classification) and some cross-validated probability estimates of one of the classes. Try e.g. AUC to form the ROC plots yourself.

4: Yes correlation filtering can also work. RF or GBM handle collinearity quite well but not perfect. Hoards of redundant or unrelated features will lower accuracy. In practice you may find that, filtering by variable importance is sufficient. As a thumb-rule, two completely redundant variables tend to share the same raw unscaled variable importance. Thus, filtering by variable importance is automatically also correlation filtering.

Bonus: Found this great article on nested-CV and tuning best-practice.

  • $\begingroup$ Dear Soren, thank you for your valuable answer !! Some further questions to your answers : $\endgroup$
    – Jason
    Commented Jan 8, 2016 at 9:58
  • $\begingroup$ 1) As by default for random forests (and i believe for boosted trees the OOB is reported in the final model--but only CV, repeated CV etc is used for the selection of the final model based on ROC metric, should i then use something like that with the full dataset ? i can post the code in order to be more clear !! Regarding your second question(in conjuction with the last question)--thus i can use all my predictors-genes, and based for instance of RFE to keep only the "best predictors ? And finally, regarding answer3: i can use the total dataset as training, and then compute the AUC ? $\endgroup$
    – Jason
    Commented Jan 8, 2016 at 10:09
  • $\begingroup$ 1) If you perform tuning and variable selection, the inner loop OOB-CV is over optimistic. You can report the average innner loop OOB-CV, but I would not put too much emphasis on that number. Set ntree high anyways. Use outer loop to estimate future model performance. The same goes for OOB-CV for the final full model, nice to know, but not the gold-standard estimate of future prediction performance. $\endgroup$ Commented Jan 8, 2016 at 10:34
  • $\begingroup$ @*Regarding your second question(in conjuction with the last question)--thus i can use all my predictors-genes, and based for instance of RFE to keep only the "best predictors ?* - I guess, I would make a model object performing some prevariable selection defined by some hyper parameters and then tune these hyper paremeters together with all other parameters as a part of the inner loop grid search. $\endgroup$ Commented Jan 8, 2016 at 10:41
  • $\begingroup$ i see-maybe as a starter i don't fully "capture" your indication of outer loop, because im using caret with functions trainControl and train which handle all the computations---regarding the number of tree, im considering even using like 10,001 or such number--maybe i could expand the grid on some parameters---moreover, i also guess than finding also independent datasets with similar design is crusial for the independent testing $\endgroup$
    – Jason
    Commented Jan 8, 2016 at 10:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.