Dataset size is very small. Sample size is 100.
I try to make machine learning model which can predict disease status.
What i have done:
- train-test split
- hyperparameter tuning using 5-fold cross-validation, with train data only.
- training model using best h-param set.
- evaluate the performance with test data.
I got acceptable AUC(0.8) using above scheme.
But my colleague said "you should test the model's performance several times with multiple train-test split. (e.g. python StratifiedShuffleSplit)"
So, Do i need to split dataset several times and apply the above scheme for each split data?