I have a general question regarding model evaluation for random forest with low sample size and unequal class distribution.

I am doing some explorative modeling by using 400 features to classify patients into two groups. The features are self-assessment items as well as change on these items during the early phases of treatment.

The total n is 260 with 31 patients in the minority class (12 %).

I am using 10- fold cross validation in the training step. I have chosen AUC as my major performance measure and doing subsampling during resampling with “rose” in the training step.

The best model from the training step achieved auc = 0.72 and preformed auc = 0.76 (with sensitivity .86 and specificity .72) with the test sample (which contained 25% of the data)

According to other studies AUC values of .70 and higher would be considered strong effects for this problem.

My question is can I trust these results considering the unequal class distribution and low sample size? I am aware that this model most likely does not generalize well to other samples but could it be used as a proof of concept or to generate hypothesis about how to detect this group of patients early in treatment (these features has not been used in other studies to detect this group of patients and it might contribute to better understanding of the problem)? Or is there something fundamentally wrong with this approach that I need to consider? Might there be a better approach?

PS I have tried different feature selection procedures like Boruta, learning vector quantization and recursive feature elimination resulting in a considerable drop in auc (however there might be ways to tweak these approaches better considering the unequal class distributions)

  • 2
    $\begingroup$ If you do get an AUC of 0.76 with CV which is roughly similar to your training AUC of 0.72 then this is a good result that your model has learned something. You may have had better results with different approaches but otherwise it looks good. $\endgroup$ – user2974951 Dec 14 '18 at 11:00

The rest seems to be fine, but 31 patients is a very small sample for minority class, to generalize the results. If you split this to train and test sets, this gets even smaller. With such a small set, it is questionable if you should split the data. After splitting the data, both training sample is too small for training, and test set too small for validating the results. Think of the test set size, is it 15 patients from the positive class, maybe 5 of them?

Better use whole data and a method that is less likely to overfit, that is, something simple, like logistic regression, but random forest with shallow trees, and prohibiting the small splits, should also do fine. You also should try reducing the number of features, because with more features then samples, risk of overfitting is greater (certain in many cases).

Then conduct detailed analysis of in-sample fit, that looks at the distributions of predictions, variable importances, tries to understand the predictions, detect biases etc. With such a small sample, you could check the predictions case by case (all of the 31 patients, plus random sample from negative class), to see what has lead the algorithm to make it's decisions and if they they make sense, or are there signs of overfitting (nonsense patterns).

  • $\begingroup$ I did a rerun of the model using only the 10 most important features from the full model (with 400 features) and got similar results. Does this add any value to the validity of my model or is it redundant becuse of data leakage? $\endgroup$ – Carl Apr 1 at 7:33

Look at the variable importance for of your training. If a single feature is much higher important than the others you need to examine is any data leak there. If not, then the results should be ok. Random Forest, and decision trees in general, are more sesitive to local (conditional) distribution rather than overall distribution.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.