I just came across a biokinesiology paper that used some Machine Learning methods, but I think there is a flaw in their methodology.

The authors had data on stroke patients and used Lasso regression to find what features about stroke patients are the best predictors of their post-stroke motor-function recovery. They varied their hyper-parameters (lambda for Lasso, whether polynomial degrees of features were used or not) and used cross-validation (CV) to test a number of different models and find which one performs best. After picking the best-performing model from CV, the authors proceeded to make inferences from this Regression model, concluding that certain features about patients are the best predictors of their recovery.

However, I would think that before making such inferences, one should have to evaluate the model performance first on unused test data to ensure that it's still a good model. If the authors are using 20-fold CV to find a good model, I would think one of these models could perform well simply from chance. So one should first evaluate this model on a test-set to ensure that the model still makes predictions with low errors, and then start to make any inferences from the model. Is my reasoning correct?

  • $\begingroup$ Can you please link the paper? On face value, if we do not have the luxury of a large enough sample using repeated $k$-fold CV is not wrong to estimate performance generalisation. (I leave causality claims aside.) 20-fold CV is indeed unusual but not a priori wrong. (+1) $\endgroup$ – usεr11852 Apr 21 '19 at 10:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.