# Before using CV-selected Regression model for Inference, shouldn't model performance be evaluated on unused test set?

I just came across a biokinesiology paper that used some Machine Learning methods, but I think there is a flaw in their methodology.

The authors had data on stroke patients and used Lasso regression to find what features about stroke patients are the best predictors of their post-stroke motor-function recovery. They varied their hyper-parameters (lambda for Lasso, whether polynomial degrees of features were used or not) and used cross-validation (CV) to test a number of different models and find which one performs best. After picking the best-performing model from CV, the authors proceeded to make inferences from this Regression model, concluding that certain features about patients are the best predictors of their recovery.

However, I would think that before making such inferences, one should have to evaluate the model performance first on unused test data to ensure that it's still a good model. If the authors are using 20-fold CV to find a good model, I would think one of these models could perform well simply from chance. So one should first evaluate this model on a test-set to ensure that the model still makes predictions with low errors, and then start to make any inferences from the model. Is my reasoning correct?

• Can you please link the paper? On face value, if we do not have the luxury of a large enough sample using repeated $k$-fold CV is not wrong to estimate performance generalisation. (I leave causality claims aside.) 20-fold CV is indeed unusual but not a priori wrong. (+1) – usεr11852 Apr 21 '19 at 10:59