I seek a rigorous or otherwise proof for the need of verification/validation data. Typically it is suggested, after building a model, to do any validation of the model's health on a set of data that was external/outside of the training/calibration data set.

While this make some intuitive sense, because there are unforeseen biases that could exist in the training set, that is not a strong enough reason on its own.

For what its worth, I typically work in the regime of parameter selection (that is, my model's form is fixed (and not at all linear), and I am only seeking to fit the values of various parameters in the model).

Also valid is an explanation as to how bootstrapping the calibration data set into a validation data set is a valid replacement for a true external data set (assuming that is the case).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.