Random forest k-fold cross validation metrics to report

Question

I am building a classifier using random forest. I have separated my data to 80% training set and 20% test set.

I did the cross-validation for feature selection, and looked at the OOB error rate to choose the number of trees. I used the model I got on the test set and got the TPR, TNR, PPV and so on.

What I want is to do a k-fold cross validation on my training set to make sure I'm not overfitting, but I'm not sure which metrics should I look at. I have the AUC for each fold, should I just look at it's average and STD? Are there other parameters I could use to show how general my model is? Is it even necessary to show that?

Answer 1

Doing an outer loop of cross validation around your random forest does make sense in a number of situations, e.g. if your data is clustered such as containing repeated measurements. The key there is to set up the cross validation so that you get out-of-(suspected)-cluster predictions.

I'm not sure which metrics should I look at

You need to look at the metrics that are important for your application, both for out-of-bag and cross validation.

should I just look at it's average and STD

I'd recommend repeated/iterated $k$-fold cross validation. With that you can measure the stability of the predictions for the same test case wrt. slight changes in the training data. Aggregation helps to improve this stability, so IMHO it should be checked afterwards.