I seek a rigorous or otherwise proof for the need of verification/validation data. Typically it is suggested, after building a model, to do any validation of the model's health on a set of data that was external/outside of the training/calibration data set.
While this make some intuitive sense, because there are unforeseen biases that could exist in the training set, that is not a strong enough reason on its own.
For what its worth, I typically work in the regime of parameter selection (that is, my model's form is fixed (and not at all linear), and I am only seeking to fit the values of various parameters in the model).
Also valid is an explanation as to how bootstrapping the calibration data set into a validation data set is a valid replacement for a true external data set (assuming that is the case).
