I'm using 10-fold cross validation to make binary classifier.
My dataset contains 3000 samples with only 150 in the minor class (low signal 4%)
I've have around 100 features and use features selection (on variance, mean difference,...) in order to reduce the number to 25.
The ratio number fo samples/number of features is 120 for the whole dataset but only 6 for the signal (minority class)
Anyway, i'm using randomforest and k-Fold Cross-Validation (k=10), and also a variable threshold (Usualy 0,5 mut in my case close to 0,11) to maximize F1-score (Harmonic mean between precision and recall).
Should hold out a test set to check the results,for example 10% which means only 15 sample of the minority class! and 90% for the train/validation part ?
The first try give me bad results (F1score close to 0,2), do i ve to iterate ? how many times ? (i.e loop on my process and mean all F1score of my tests).
Even on randomforest parameters it seems that number of iteration and deep size always increase for better results.
I've read to limitate those parameters in order to avoid overfitting , is that correct ?
Thank's for your help