# Is using a test set mandatory after a k-fold cross-validation?

I'm using 10-fold cross validation to make binary classifier.

My dataset contains 3000 samples with only 150 in the minor class (low signal 4%)

I've have around 100 features and use features selection (on variance, mean difference,...) in order to reduce the number to 25.

The ratio number fo samples/number of features is 120 for the whole dataset but only 6 for the signal (minority class)

Anyway, i'm using randomforest and k-Fold Cross-Validation (k=10), and also a variable threshold (Usualy 0,5 mut in my case close to 0,11) to maximize F1-score (Harmonic mean between precision and recall).

Should hold out a test set to check the results,for example 10% which means only 15 sample of the minority class! and 90% for the train/validation part ?

The first try give me bad results (F1score close to 0,2), do i ve to iterate ? how many times ? (i.e loop on my process and mean all F1score of my tests).

Even on randomforest parameters it seems that number of iteration and deep size always increase for better results.

I've read to limitate those parameters in order to avoid overfitting , is that correct ?

• The question is hard to understand. Please start with correcting grammar and punctuation to make the question legible. Then, please give us more details: (i) what is the F1 score? (ii) what does it mean "try give me bad results"? (iii) what do you mean by "do i ve to iterate"? (you already do CV?) – January Nov 22 '18 at 10:50

For a 3000 samples dataset, I would suggest a 70-30 split to determine training and validation datasets. Even though you are taking advantage of k-fold cross-validation, the verifications run with a test data made of "unseen" samples is mandatory.

To compensate for the unbalance, you may take advantage of one of the following techniques:

• down sampling the majority class
• up sampling the minority class
• SMOTE

If you are using R, you may take advantage of caret package facilities to achieve that. caret package also allows to specify tuning grids and plot the classifier performance. Use something as:

trControl <- trainControl(method = "repeatedcv",
number = 10,
repeats = 5,
verboseIter = FALSE,
sampling = "up",
classProbs = TRUE,
summaryFunction = twoClassSummary,
savePredictions = TRUE,
returnResamp = "all")

rf_fit <- train(
frm,
data = train_dataset,
method = "rf",
trControl = trControl,
metric = 'ROC',
tuneLength = 10
)
plot(rf_fit)


The sampling parameter can be:

• sampling = "up"
• sampling = "down"
• sampling = "SMOTE"

and take advantage of the ROCR package facilities to plot the ROC in order to choose a decision threshold to tune required specificity and sensitivity metrics.

See this example.

• Thanks for your advice, i've allready did undersampling of the majority class (initialy 10 000) I'll try oversampling of minority one. Do you think i've to loop on my CV+Test process in order to average the F1score of each loop ? – Fabrice JOURDAN Nov 24 '18 at 11:46
• You have to average the metrics set obtained at each step of cross-validation, Hence collect F1score for each cross-validation loop and then average them at the end of CV. If you have multiple test set, you may want to average any evaluation metrics. – GiorgioG Nov 30 '18 at 17:14
• Also consider the minimum and maximum values for each metrics you collect in CV to determine intervals of values of. – GiorgioG Dec 1 '18 at 9:24