Split-sample validation is always controversial unless you have enormous training and test samples. You might consider improving the model at will and doing a rigorous bootstrap validation at any point. Also consider putting smooth time trends in the model. As shown in my RMS book and notes (see http://www.fharrell.com/p/blog-page.html) you are somewhat safe to be making the model more complex in an honest fashion, because adding parameters results in confidence limits for predictions that are self-penalizing (confidence bands get wider as you add more parameters, but narrower as you add more data). But most practitioners fail to show the confidence bands (not a good idea). Also consider the heuristic slope shrinkage estimator gamma-hat which estimates the slope of the calibration plot on the linear predictor scale. It is a function of the model likelihood ratio chi-square statistic and the number of candidate parameters entertained or fitted.

Split-sample validation is always controversial unless you have enormous training and test samples. You might consider improving the model at will and doing a rigorous bootstrap validation at any point. Also consider putting smooth time trends in the model. As shown in my RMS book and notes (see https://hbiostat.org/rms) you are somewhat safe to be making the model more complex in an honest fashion, because adding parameters results in confidence limits for predictions that are self-penalizing (confidence bands get wider as you add more parameters, but narrower as you add more data). But most practitioners fail to show the confidence bands (not a good idea). Also consider the heuristic slope shrinkage estimator gamma-hat which estimates the slope of the calibration plot on the linear predictor scale. It is a function of the model likelihood ratio chi-square statistic and the number of candidate parameters entertained or fitted.