Evaluate Sample Representativeness of Dataset

Question

I'm trying to check / evaluate how representative a sample is from a dataset.

I'm interested in two things:

1. Which sample best represents the original dataset
2. Is the sample a "good enough" representative of the dataset

Example scenario: I have a dataset

original:

> head(mtcars,n=10)
                   mpg cyl  disp  hp drat    wt  qsec vs am gear carb
Mazda RX4         21.0   6 160.0 110 3.90 2.620 16.46  0  1    4    4
Mazda RX4 Wag     21.0   6 160.0 110 3.90 2.875 17.02  0  1    4    4
Datsun 710        22.8   4 108.0  93 3.85 2.320 18.61  1  1    4    1
Hornet 4 Drive    21.4   6 258.0 110 3.08 3.215 19.44  1  0    3    1
Hornet Sportabout 18.7   8 360.0 175 3.15 3.440 17.02  0  0    3    2
Valiant           18.1   6 225.0 105 2.76 3.460 20.22  1  0    3    1
Duster 360        14.3   8 360.0 245 3.21 3.570 15.84  0  0    3    4
Merc 240D         24.4   4 146.7  62 3.69 3.190 20.00  1  0    4    2
Merc 230          22.8   4 140.8  95 3.92 3.150 22.90  1  0    4    2
Merc 280          19.2   6 167.6 123 3.92 3.440 18.30  1  0    4    4

and I take 4 samples, sample1 will have 20% of the records, sample2 will have 40%, sample3 will have 60% and sample4 will have 80% of the original dataset (taken at random).

I want to be able to compare these 4 samples with the original and show how their representativeness changes as the percentage changes, even though, all could be good and bad representatives, the larger samples should have a larger chance of being representative (ie I will run the test multiple times).

I have tried using RMSE, MASE, Chi Sq test and a couple of other methods in R but with no luck.

Any and all help will be greatly appreciated!

Answer 1

In a sense, all samples are representative if they're suitably randomly chosen. However, I think you are more interested in computing a measure of a sample's closeness to a population.

In principle you:

1. Decide a set of sample statistics that you want to be close to their population versions, e.g. a) means and variances of each variable, b) means, variances, and covariances between the variables, or c) all these and some higher moments too, etc.

2. you define a distance metric that reflects them. As an example, something like Mahalanobis distance will do c. for you, off the shelf.

3. Draw a sample and apply the metric

For more inspiration you might want to look at what statistics are used in the balance tests for matching.

Answer 2

If you're OK with the assumption that the population roughly follows a multivariate normal distribution, you could do the following:

1. Estimate the parameters of the distribution from your sample (mean and covariance matrix)
2. Compute the likelihood of the population data based on those parameters (multivariate normal probability density function)
3. Compare the population's likelihood based on parameters from different sample sizes