So basically, we have a sample of known data taken from a larger sample of known data. Is there a good way to measure how "representative" that data is, especially in distributional terms, compared to the larger sample from which it was taken? I'm basically looking for a specific metric that compares their distributions, and I would theoretically add data from the larger sample into the smaller and the metric should say there is an improvement.

It seems like the Kolmogorov–Smirnov test is what I am looking for more-or-less.

The use case is if I want a small representative sample from a large store of data to shove in a model without hold-out testing to see how that model generalizes--yes that's bad but that's this specific use case. Because of this limitation, I have to know that the smaller sample I want to put in the model is representative of the larger dataset.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.