1
$\begingroup$

I seek a rigorous or otherwise proof for the need of verification/validation data. Typically it is suggested, after building a model, to do any validation of the model's health on a set of data that was external/outside of the training/calibration data set.

While this make some intuitive sense, because there are unforeseen biases that could exist in the training set, that is not a strong enough reason on its own.

For what its worth, I typically work in the regime of parameter selection (that is, my model's form is fixed (and not at all linear), and I am only seeking to fit the values of various parameters in the model).

Also valid is an explanation as to how bootstrapping the calibration data set into a validation data set is a valid replacement for a true external data set (assuming that is the case).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.