I'm modelling a continuous variable (say, the average amount of smth per client). The variable has some asymmetric distribution: for example Gamma/Tweedie/ etc.

Suppose that I'm not able to do cross validation after building a model: All I can do is to select train/test subsets once (80%/20%) from initial dataset and then train model using train set.

The problem is that when generating 80% using pseudo random variable it might happen that my train test does not correctly resemble original dataset. Also the problem is that train and test set could not resemble each other.

Does anyone know how to correctly split data into train/test so that each part of train/test would resemble each other and initial distribution?

I understand that usually I should use cross-validation while selecting model parameters to overcome such type of problems, but is there anything one could do without it? I found some information about KLIEP algorithm but I'm not sure that it is applicable to the case mentioned above.

I would appreciate any comments/links to read.

  • 2
    $\begingroup$ You can always stratify your sample so that the distribution of the underlying variables is similar between the two sets. There are techniques such we do precise covariate balancing between control and treatment groups that could also be applicable but they are almost certainly an over-kill to use in this setting. $\endgroup$
    – usεr11852
    Jan 30 '19 at 23:25
  • $\begingroup$ @usεr11852 could you provide a reference how to do it in case of continuous response, please? Or maybe expand your comment a bit $\endgroup$
    – xxxxx
    Jan 30 '19 at 23:52
  • $\begingroup$ I was actually doing that as I realised it was a bit too scarce! :) Please see my answer below. $\endgroup$
    – usεr11852
    Jan 30 '19 at 23:53

We can always stratify our sample so that the distribution of the underlying variables is similar between the two sets; stratified sampling is quite standard approach to ensure random subgroups have similar statistical properties. If we are using R they are multiple packages offering stratified sampling; e.g. the packages splitstackshape and stratification have a lot of readily available functionality. Most of stratified sampling methodology originates from survey statistics and ecology, so one might want to see a paper like Shao's (2003) "Impact of the Bootstrap on Sample Surveys", to get a better idea about potential implications of bootstrapping a (survey) sample. I have also found the UN's FAO (Food and Agriculture Organisation of the United Nations) Fisheries Technical Paper 434 on Sampling methods applied to fisheries science: a manual extremely readible and to the point (see in particular section 4 "Stratified random sampling")

There are techniques that allows precise covariate balancing between control and treatment groups that could also be applicable but they are almost certainly an over-kill to use for picking a hold-out set. They might be useful as diagnostic tools nevertheless.

  • $\begingroup$ Correct me if I misunderstood smth please. So, Basically the idea is following: I should bin my continuous variable (into large enough amount of bins), call the stratified sampling function on my response and thats it? The nature of stratification guarantees that empirical distributions will be similar? $\endgroup$
    – xxxxx
    Jan 31 '19 at 0:18
  • $\begingroup$ 1. Generally we would like to avoid the binning of a continuous variable as it may create discretisation artefacts; that said, it is a quick-and-dirty solution for what you want (a proper solution would be a moment-matching sampling approach). 2. Yes, you are correct; the idea of stratification is to guarantee that the empirical distributions will be similar. $\endgroup$
    – usεr11852
    Jan 31 '19 at 0:42
  • $\begingroup$ @ usεr11852 Would you be so kind also to give a hint/reference to method how could I ensure that two sets are belong to same distribution? Is there any statistical test? Maybe I should somehow compare distributions via Kolmogorov-Smirnov test? Or maybe is it just enough to compare means and variances (I personally don't think that it is sufficient)? $\endgroup$
    – xxxxx
    Jan 31 '19 at 20:43
  • $\begingroup$ @ usεr11852 I had two ideas: 1) We can compare means and variances of Empirical Distribution Functions (or smith similar) for each of 3 sets: original/train/test 2) We can calculate empirical distribution for original set and use KS to make sure that it fits our train/test. Is any of these makes sense? Or maybe there is any other "state of art" solution to question? Sorry if I'm "statistically" incorrect, I'm very new to this field. $\endgroup$
    – xxxxx
    Jan 31 '19 at 21:04
  • $\begingroup$ Your ideas are perfectly fine. I would suggest you look into the article: [Reporting of covariate selection and balance assessment in propensity score analysis is suboptimal: a systematic review] by Ali et al. as it has a full list of methods assessing covariate balance. In general, for a large original set, one would not expect massive differences between a randomly chosen split unless dealing with pathological cases, or very imbalanced data. $\endgroup$
    – usεr11852
    Jan 31 '19 at 22:00

In addition to @usεr11852's answer:

  • if you can do train/test splits that are random but have the desired distribution matching properties, you can repeat/iterate this: this is a resampling technique called set validation and in this is similar to cross validation and out-of-bootstrap validation.

  • For a single continuous variable of interest, there are non-random cross validation schemes like venetian blinds splitting where you sort your samples according to the continuous variable and then assign groups with equal sample no. modulo $k$. That leaves a slight systematic difference between the groups.

  • For multiple such covariates, clustering methods like k-means (using what is otherwise mostly considered a weakness: that k-means tends to produce similar-sized spherical clusters) or Kohonen maps have been used for the stratification.

  • The Duplex algorithm (Snee: Validation of Regression Models: Methods and Examples, Technometrics 1977 (a variant of the Kennard-Stone algorithm) gives a single split into train and test sets.
    This algorithm is particularly interesting if you need to select a subset of a larger number of data points for training and testing (e.g. lots of measurements available, but reference values are expensive).

  • $\begingroup$ +1 Nice resources, I will definitely look at that article! $\endgroup$
    – usεr11852
    Jan 31 '19 at 22:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.