I have a general question regarding model evaluation for random forest with low sample size and unequal class distribution.
I am doing some explorative modeling by using 400 features to classify patients into two groups. The features are self-assessment items as well as change on these items during the early phases of treatment.
The total n is 260 with 31 patients in the minority class (12 %).
I am using 10- fold cross validation in the training step. I have chosen AUC as my major performance measure and doing subsampling during resampling with “rose” in the training step.
The best model from the training step achieved auc = 0.72 and preformed auc = 0.76 (with sensitivity .86 and specificity .72) with the test sample (which contained 25% of the data)
According to other studies AUC values of .70 and higher would be considered strong effects for this problem.
My question is can I trust these results considering the unequal class distribution and low sample size? I am aware that this model most likely does not generalize well to other samples but could it be used as a proof of concept or to generate hypothesis about how to detect this group of patients early in treatment (these features has not been used in other studies to detect this group of patients and it might contribute to better understanding of the problem)? Or is there something fundamentally wrong with this approach that I need to consider? Might there be a better approach?
PS I have tried different feature selection procedures like Boruta, learning vector quantization and recursive feature elimination resulting in a considerable drop in auc (however there might be ways to tweak these approaches better considering the unequal class distributions)