Currently, I am encountering a question, which is how to selection representative samples (training set and test set, even validation set) from the whole data set? I would like build a classification model using a training set and adjust the parameters using a validation set, and at last do prediction to the external test set which is not used to build model (in practice, some unseen data without labels).
Usually, some common methods are cross-validation, hold-out, etc. Random split methods can deal with the problem. But, when further data points do not fall in the model domain, the prediction will be not reliable. So instead of randomly splitting into training and test sets, are there split methods based on the independent variables (X), other than the Kennard-Stone algorithm? Or are there any algorithms can do these things reasonably? (These algorithms should consider the sample distribution between training and test set.)