I'm most (and most frequently) annoyed by "validation" aiming at generalization error of predictive models where the test data is not independent (e.g. typically multiple measurements per patient in the data, out-of-bootstrap or cross validation splitting measurements not patients).
Even more annoying, papers that give such flawed cross validation results plus an independent test set that demonstrates the overoptimistic bias of the cross validation but not a single word that the design of the cross validation is wrong ...
(I'd be perfectly happy if the same data would be presented "we know the cross validation should split patients, but we're stuck with software that doesn't allow this. Therefore we tested a truly independent set of test patients in addition")
(I'm also aware that bootstrapping = resampling with replacement usually performs better than cross validation = resampling without replacement. However, we found for spectroscopic data (simulated spectra and slightly artificial model setup but real spectra) that repeated/iterated cross validation and out-of-bootstrap had similar overall uncertainty; oob had more bias but less variance - for rewieving, I'm looking at this from a very pragmatic perspective: repeated cross validation vs. out-of-bootstrap does not matter as long as many papers neither split patient-wise nor report/discuss/mention random uncertainty due to limited test sample size.)
Besides being wrong this also has the side effect that people who do a proper validation often have to defend why their results are so much worse than all those other results in the literature.