Important questions regarding the methodology for constructing classifiers with R package caret and tree based algorithms

Question

I'm currently playing with caret R package, with one merged microarray affymetrix dataset(paired tissue samples), in order to build and test various classifiers, mostly based on trees-such as random forests and boosted trees to predict a binary outcome-categorical, that is disease status (normal and cancer status). In detail, my main "introductory" questions are the following:

1) Because the sample size of my dataset is somehow described as "medium"-60 samples(maybe even small), should i split my dataset like 60% training and 40% test set ? Or is more appropriate especially for random forests, because of the out of bag error-OBB( because is like an independent "testing set") to use all my dataset, and test independently in other dataset(s) with similar design/status ?

2) Moreover, except my genes-predictors, i also have 8 continuous PET variables, which also describe the same patients from my specific dataset. Thus, my main interest, is if the addition of these PET variables improve the outcome/prediction. An illustrative example of my PET variables is as follows (samples in the rows, PET variables in the columns):

Thus, after I construct a total data.frame of both genes and these PET variables in the columns, should i also transform-"preprocess" my total variables, like centering or scaling ? Or for the mentioned classification algorithms it does not matter, because RF can handle the "different nature" of the variables ? and the same applies with gbm method ? And overall is it correct to use different variables with RFs ?

3) Moreover, in case I want to build a ROC curve, to see the performance of the classifier based on the training set, how could I use similarly the pROC package from the below caret tutorial ?

http://static1.squarespace.com/static/51156277e4b0b8b2ffe11c00/t/5310d223e4b0d21c529a0814/1393611299730/webinar.pdf

In detail, from the above tutorial, the object gbmtune is the output of the train function. How could I use it similarly with the roc function, in order to create the ROC curve?

4) Finally (one more general question not explicitly related to my specific dataset), because in one other dataset I have a great number of predictors, is it appropriate to use first a correlation filter to remove redundant features, and then continue with the RFE methodology to "identify" the "best performers?

Please excuse me for any naive questions, but there are fundamental in order to have an appropriate approach on this task!

Answer 1

1: Yes you should either use OOB-CV with 5000 trees and/or 10-fold 20-repeated. Training on only 60% is likely to yield a pessimistic cross validation. With such few samples, any single CV will vary considerably. Repeat the 10-fold CV many times and compute the average score(+/- SD). Do not pool predictions from repeated CV and then after compute the statistics. That will yield a over optimistic CV. If using OOB-CV only, then train 5000-10000 trees to achieve a stable result. After CV, use all samples to train a single model.

2: Any decision tree model, such as random forest and gbm (gradient boosting machine, uses trees as default) handle features as non-parametric and is invariant to any monotonic transformation. Thus, no scaling etc. is needed. It's a fine start just to mix all features from both sources. You may want to perform some modest variable selection on your features, as most out of 5000 micro-array features are expected to be less useful. Don't search for a magic perfect combination of ~7 variables, but narrowing the field of potential useful variables may yield a modest improvement. Try e.g. to pretrain a RF and drop ~50-90% of variables with lowest permutation variable importance (never use gini). Variable selection needs to be cross-validated also. The direct OOB-CV error after variable selection is over optimistic. I was in similar situation training a RF model with 41 samples(very few) and 250 features(reduced to 30 features). Variable selection seemed to yield a modest prediction accuracy improvement in outer cross-validation and in external test sets.

3: For any classifier which yields not only labeling, but also yields some probability estimate, can be used to form a ROC plot. A ROC-plot communicates any attainable combination of selectivity and sensitivity. Here is ROC example for randomForest. R package for Weighted Random Forest? classwt option? . In the slides [slide 16], the caret package is used to tune a gbm model with a ROC metric in a inner calibration loop, [edited] but not cross-validated with a outer validation loop. Thus, the tuning only selects a good set of hyper-parameters but do not provide an unbiased CV estimate. For that, wrap the tuning in a outer cross-validation manually The tuning/calibration metric does not have to match the cross-validation metric. To form a ROC plot you need the true labels(binary classification) and some cross-validated probability estimates of one of the classes. Try e.g. AUC to form the ROC plots yourself.

4: Yes correlation filtering can also work. RF or GBM handle collinearity quite well but not perfect. Hoards of redundant or unrelated features will lower accuracy. In practice you may find that, filtering by variable importance is sufficient. As a thumb-rule, two completely redundant variables tend to share the same raw unscaled variable importance. Thus, filtering by variable importance is automatically also correlation filtering.

Bonus: Found this great article on nested-CV and tuning best-practice.