Currently, I am encountering a question, which is how to selection representative samples (training set and test set, even validation set) from the whole data set? I would like build a classification model using a training set and adjust the parameters using a validation set, and at last do prediction to the external test set which is not used to build model (in practice, some unseen data without labels).

Usually, some common methods are cross-validation, hold-out, etc. Random split methods can deal with the problem. But, when further data points do not fall in the model domain, the prediction will be not reliable. So instead of randomly splitting into training and test sets, are there split methods based on the independent variables (X), other than the Kennard-Stone algorithm? Or are there any algorithms can do these things reasonably? (These algorithms should consider the sample distribution between training and test set.)


If you want to go this route, you will need to consider the structure within your data in deciding which instances to use at some point (e.g. build a model).

One way to do this would be clustering your data and then using some instances from every cluster (depending on its size). If the clustering method and the classification method are related (for example both kernel-based), you are probably implicitly causing information leaks in the sampling approach. Ofcourse, the clustering method you choose essentially imposes a lot of assumptions so whether or not this beats random is very much the question.

Personally, I prefer randomized approaches: few assumptions and they never failed me yet in practice.

  • $\begingroup$ Thanks for your response. I need understand more. But, yes, Random selection is more popular. $\endgroup$ – Kevin Oct 13 '14 at 3:26
  • 1
    $\begingroup$ However, if you have an underlying structure (= clusters) in the data, you'd need to test on left out clusters. Comparing the results of excluding some cases from each cluster for testing and excluding whole clusters for testing may give valuable information whether such a practically relevant clustered structure is in the data. $\endgroup$ – cbeleites unhappy with SX Oct 13 '14 at 13:13

It seems to me that you are thinking about making and using a single partition of your data for training and validating your model. It is true that the ability of cross-validation to yield a good model can depend on the quality of the partition, and that a random partition can be poor. I would still use random partitions as @MarcClaesen suggests, but try doing many of them and averaging over them. With enough iterations, the process should work quite well, even if there is no guarantee any individual partition was perfectly representative.

  • $\begingroup$ Thanks gung. I can get the good enough models using 30 times of held-out predictions. when using this model to predict the unlabelled data, the results give more positive results. I can not believe it. So I am think if the unseen data points are distributed in the training data points. $\endgroup$ – Kevin Oct 14 '14 at 2:02
  • $\begingroup$ When @gung mentions 'doing many of them and averaging over them.' I don't know if I am being slow but what is meant by averaging over them? $\endgroup$ – DataJack Feb 7 '17 at 15:11
  • $\begingroup$ @DataJack, it may help you to review how cross-validation works. Often you partition a dataset into k subsets & fit your model k times, checking against each of the k holdout sets in turn. To assess the out of sample performance, you average the performance on the k holdout sets. You can also repeat that entire process l times & average over the lxk holdout sets. $\endgroup$ – gung - Reinstate Monica Feb 7 '17 at 15:34
  • $\begingroup$ @gung Apologies I thought you meant extract samples and before undertaking CV average over them. I thought you were implying creating an 'average' dataset from all of the samples. $\endgroup$ – DataJack Feb 7 '17 at 15:38
  • $\begingroup$ @DataJack, I'm not sure if this is what you're after, but the antecedent of "average over them" is "random partitions as MarcClaesen suggests". $\endgroup$ – gung - Reinstate Monica Feb 7 '17 at 15:42

You did not specify your sample size and distribution of $Y$. Depending on that, the size may be insufficient for reliably doing any splitting method other than 100 repeats of 10-fold cross-validation. The bootstrap may be your best bet unless the total sample size is huge. And what makes you choose a classification method or a prediction method? No matter what you do, avoid at all costs the use of improper accuracy scoring rules such as the proportion 'classified' correctly. This throws out much of the information in your sample.

  • $\begingroup$ Thanks Frank. I have 204 negative samples (non-interact) and 635 positive samples (interact) as well as 839 samples without labels. Initially, I build the Random forest model using the labelled data, adopting the default setting in RF, resulting a very good performance in terms of sensitivity, specificity, F score and MCC for the test set (labelled data). However, when predicting the unlabelled data, I found the model predict most of the unlabelled data having interactions. I can not believe the results. can you give some suggestions for this results? $\endgroup$ – Kevin Oct 14 '14 at 1:56
  • $\begingroup$ You didn't get my point about avoiding improper scoring rules such as classification accuracy, sensitivity, specificity. With classification accuracy if $Y=1$ a fraction $z$ of the time you will be right $z$ of the time by predicting every observation to have $Y=1$. Choose a method that provides predicted probabilities and use a proper accuracy scoring rule as well as a semi-proper one such as concordance probability (ROC area). Classification involves arbitrary thresholds on probability forecasts. $\endgroup$ – Frank Harrell Oct 14 '14 at 2:51
  • $\begingroup$ Thanks Frank. Sure I will try some methods which can calculate the predicted probabilities, not just predicted labels, and select the proper cutoff to check the performance. $\endgroup$ – Kevin Oct 14 '14 at 14:08
  • $\begingroup$ Proper assessment of performance uses no cutoffs whatsoever. $\endgroup$ – Frank Harrell Oct 14 '14 at 15:52
  • $\begingroup$ what is such proper statistical metric? ROC area (AUC)? if the class is imbalanced, AUPR is more proper? is there other more proper metrics for assessing the performance of model? Thanks in advance! $\endgroup$ – Kevin Oct 14 '14 at 16:46

I typically also use the randomized approaches. That being said, depending on your application/data structure and accessibility of cases it may be worth while to look at additional splits that give you an idea about the ability to extrapolate to cases outside the training domain.

My reasoning behind this is that IMHO you can use your model for whatever data domain (theoretically independent of training domain - in practice it will often be closely related to the training domain) as long as you can demonstrate that the predictive ability is good enough for that domain.

I'm chemometrician, so I'll formulate some chemistry-related example scenarios.

  • Consider process analysis. You may receive both data from typical batches and untypical batches. In this scenario it would be important to measure the performance both for unknown typical and unknown untypical batches (separately). Maybe even the performance for untypical batches if the model is trained on typical batches only.

  • Consider a medical diagnostic problem, and data for a number of patients. Typically, you'd randomize patients. However, you may in addition look at the performance for test patients which are in addition extreme according to some other parameters (features). E.g. assume one of your features is blood glucose. It may be worth while to test unknown patients with unusual (high/low) blood glucose.

  • If you are into QSAR (quantitative structure activity relationship) models, consider testing the performance not only with unknown substances but for unknown substance classes.

  • and so on

Note that you can also do this by clustering your data and then doing a leave-cluster-out validation.

  • $\begingroup$ Thanks cbeleites. I will try leave-cluster-out validation. $\endgroup$ – Kevin Oct 14 '14 at 1:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.