1
$\begingroup$

Dataset size is very small. Sample size is 100.

I try to make machine learning model which can predict disease status.

What i have done:

  1. train-test split
  2. hyperparameter tuning using 5-fold cross-validation, with train data only.
  3. training model using best h-param set.
  4. evaluate the performance with test data.

I got acceptable AUC(0.8) using above scheme.

But my colleague said "you should test the model's performance several times with multiple train-test split. (e.g. python StratifiedShuffleSplit)"

So, Do i need to split dataset several times and apply the above scheme for each split data?

Improve this question
$\endgroup$
0
$\begingroup$

K-fold cross-validation with stratified sampling will be a better approach.

The primary reason to stratify is to split them into train and test set, for each class. For example, if you have 50 samples each for class A and Class B, stratified sampling ensures 40-10 split for each class instead of 80-20 with 20 samples from same class in the test set.

The objective of using K-fold cross-validation is to shuffle the data and to cross-verify that the model didn't overfit with one subset of data. So K-fold ensures that, the model performs well for any minor subset of data if it's trained with the other major subset of data.

If its a 5 fold cross-validation, data is split into 80-20(train-test) samples each time and validated.

Improve this answer
$\endgroup$
8
  • $\begingroup$ added further explanations $\endgroup$ – venkat krishnan Jul 23 '19 at 3:49
  • $\begingroup$ If i used cross-validation in hyperparameter tuning step, are performance test of each of multiple splits not required then? $\endgroup$ – Crispy13 Jul 23 '19 at 4:30
  • $\begingroup$ not required. if you use it during hyperparameter tuning. By the way, how is the results ? Also have you checked the precision, recall, f1 score for individual folds ? if its performing well, then you can choose the best one from it. $\endgroup$ – venkat krishnan Jul 23 '19 at 4:58
  • $\begingroup$ F1-score is about 0.75. By the way, is there no possibility that a performance metric value is high by chance(lucky train-test split) if i use cross-validation? $\endgroup$ – Crispy13 Jul 23 '19 at 7:25
  • $\begingroup$ depends on luck only when you do it one split. if you do k-fold, the factor of luck is eliminated, because it gets shuffled and split differently. Good to be sceptical about good accuracy, but you can trust if it gives good result for out of the box test cases. You can also verify it by using a decision tree and verifying the variables involved in the split by checking does it actually affects the final prediction $\endgroup$ – venkat krishnan Jul 23 '19 at 7:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.