Common practice for validating classifiers in medical statistics We are writing a paper, where we suggest a new classifier. We have a data base with about 1400 medical cases.
Would it be sufficient just to divide the dataset into training 70% and test 30%? Or should bootstrap be needed? May be, 10-fold cross validation will be considered more preferable?
I am not asking about advantages and drawbacks of each procedure. I am asking about common practice only.
 A: From what I see in the current medical literature, both cross-validation or a validation based on a hold-out sample will likely get you published in OK, but not top journals (this may strongly depend on the standards in your sub-field, though). One caveat: If you are using a standard machine-learning approach (SVM etc.) you will have to perform  hyper-parameter tuning. Tuning usually requires cross-validation. You CANNOT use simple cross-validation to both select optimal hyperparameters and to validate your results. In this case use either have to rely on
a) nested cross-validation (which sounds complicated, but is just the Inception version of cross-validation ...)
or
b) validation in a hold-out sample that has not been looked at during training & parameter-tuning (30% sounds fine, given your sample size, but that depends on the performance of the classifier).
Cross-validation is arguably the "more powerful" approach (i.e. the one yielding better results), hold-out sample arguably the "cleaner" one.
However, the central point for the medical field is: "Will it help to improve the diagnosis of future cases in a real-world setting?" Therefore, the most convincing case IMHO (and the one most likely to attract top-journals): a) Train and cross-validate your model on the given retrospective data-set. b) Validate your classifier on data from a prospective validation study, that was specifically obtained for that purpose, with a pre-registered study-protocol (including the exact classifier to test), with analysts being blinded to the key validation criteria until classification, ideally covering multiple testing centers and multiple (sub-)populations, and ideally including data/tests that allow to identify mechanisms behind and potential confounders threatening the method. But this is laborious, of course.
