I'm modelling a continuous variable (say, the average amount of smth per client). The variable has some asymmetric distribution: for example Gamma/Tweedie/ etc.
Suppose that I'm not able to do cross validation after building a model: All I can do is to select train/test subsets once (80%/20%) from initial dataset and then train model using train set.
The problem is that when generating 80% using pseudo random variable it might happen that my train test does not correctly resemble original dataset. Also the problem is that train and test set could not resemble each other.
Does anyone know how to correctly split data into train/test so that each part of train/test would resemble each other and initial distribution?
I understand that usually I should use cross-validation while selecting model parameters to overcome such type of problems, but is there anything one could do without it? I found some information about KLIEP algorithm but I'm not sure that it is applicable to the case mentioned above.
I would appreciate any comments/links to read.