I just came across a biokinesiology paper that used some Machine Learning methods, but I think there is a flaw in their methodology.
The authors had data on stroke patients and used Lasso regression to find what features about stroke patients are the best predictors of their post-stroke motor-function recovery. They varied their hyper-parameters (lambda for Lasso, whether polynomial degrees of features were used or not) and used cross-validation (CV) to test a number of different models and find which one performs best. After picking the best-performing model from CV, the authors proceeded to make inferences from this Regression model, concluding that certain features about patients are the best predictors of their recovery.
However, I would think that before making such inferences, one should have to evaluate the model performance first on unused test data to ensure that it's still a good model. If the authors are using 20-fold CV to find a good model, I would think one of these models could perform well simply from chance. So one should first evaluate this model on a test-set to ensure that the model still makes predictions with low errors, and then start to make any inferences from the model. Is my reasoning correct?
