Variable selection and model validation Hey I want to build a model (choose significant variables) and validate it. Is this way correct?

*

*Divide data into train (80%) and test (20%) data

*Use train data to build a model (especially variable selection)

*When we have chosen variables, we can do k-fold validation of this model on TRAIN data

*If everything is OK, that is the results from k-fold validation are close, we can build a model on all train data and use it to check the accuracy of our model on test data

Is my way correct or I missed sth? My main point is when we select the variables for the model. And if everything is OK, should I build my model on ALL data or only on training data?
 A: 
Divide data into train (80%) and test (20%) data


*

*if you do not have separate test data then it does not make sense to divide your original data to train and test. Use k-fold cross validation instead. Test data is something that is not taken from the same population. a simple example would be data from two different hospitals H so data from H1 used for model building and H2 used for final validation.


Use train data to build a model (especially variable selection)



*Do variable selection using k-fold cv of train data so that variables are selected on train cv fold and validated on internal validation fold


When we have chosen variables, we can do k-fold validation of this
model on TRAIN data



*this is wrong as you have already chosen a variable on the entire train data, so after variable selection doing k-fold validation will include your train data in validation split and results, in this case, would be biased


If everything is OK, that is the results from k-fold validation are
close, we can build a model on all train data and use it to check the
accuracy of our model on test data



*if you do not have external test data, stop at step 2 and report internal average train and test results from k folds, if you have separate test data as explained in step 1) you can build model using the entire train data with selected variables and apply model to external test data. There can be many ways of selecting variables e.g. one way could be selecting variables that have high occurrences in train folds of step 2).

I hope this helps
