I know of random sampling and stratified sampling, but what I am not sure what is the name of the type of sampling I need:

I wish to work-out an example with my students of simple correlation/regression. I found the following nice dataset: "When do Babies Start to Crawl", but it has too many observations (12). And what I wish is something even smaller that we can do calculations on (let's say 6). I would like to sample these points, but I would rather do it in such a way so that the result of the analysis would remain as similar as possible to what I would get from the larger dataset.

So of course the SE would change, but I would like the correlation to be as close as possible, and if possible to have the range stay similar (I'm ignoring the outlier issue).

Does this type of situation have a known name which I can use when searching for good solutions?

  • $\begingroup$ When you're teaching, don't you have the liberty to just make up some data (as long as you tell the students that it's made up). Then you can pick any six observations, and tweak the numbers to illustrate whatever concept you're going for. $\endgroup$ – zkurtz Nov 3 '13 at 22:54
  • 1
    $\begingroup$ I would not call this a sample, because there is absolutely nothing random in how you select the observations (and the original sample was not random, either). I would rather refer to this as a "subset of the data". If you agree, please consider editing your title. If you don't... well, there is no way to downvote a comment ;) $\endgroup$ – StasK Nov 4 '13 at 2:06
  • 3
    $\begingroup$ @StasK There is more to the concept of a "sample" than randomization, which is a separate idea. For instance, convenience samples, judgment samples, and systematic samples are still called "samples." $\endgroup$ – whuber Nov 4 '13 at 20:23

I don't know whether it has a name, but I have used similar techniques to create synthetic datasets to answer questions on this site: they frame the problem as an optimization and then carry it out.

The methods to use for optimization depend on the problem. In this case you can explore the entire collection of possible six-element samples of the dataset, because $\binom{12}{6} = 924$ is small. In general you cannot perform an exhaustive search and have to be content with some form of randomized search, guided perhaps by methods of simulated annealing or genetic algorithms. But for your purposes you don't need a truly optimal solution, so a blind search ought to work fine.

To illustrate, I saved the "crawling" dataset in a file and applied the R script below. Its output lists the case numbers of the optimal subset and compares the statistics you would like to reproduce. (I assumed these were the coefficients of the regression of mean crawling age against temperature, but almost any small set of statistics will work provided they are not sample-size dependent, as recognized in the question itself.)

Sample: 2 3 4 5 9 12
         (Intercept) temperature
Original    35.70254 -0.07560731
Sample      35.70062 -0.07548532

In this plot of the data, the optimal subset is shown in red and the two fits in corresponding colors; they are indistinguishable.


To compare the exhaustive search to the blind random search, I set the random number seed to 17, the number of combinations to search to 924, and forced the code to perform the randomized search (thereby going to exactly the same computational effort, but with no guarantee of optimality). The output this time was

Sample: 3 5 8 10 11 12
         (Intercept) temperature
Original    35.70254 -0.07560731
Sample      35.70047 -0.075770

It is a different sample, but the results are almost as good as before.

f <- read.csv("f:/research/R/crawling.txt", sep="\t")
# Function to return the statistics to match in a sample.
get.coef <- function(g) {
  coef(lm(avg_crawling_age ~ temperature, weights=n, data=g))
# Compute these statistics for all possible samples of a specified size.
n.max <- 10^4 # Limits the execution time.
sample.size <- 6
system.time( {
  if (choose(nrow(f), sample.size) > n.max) {
    print("Using randomized search.", quote=FALSE)
    samples <- replicate(n.max, sample.int(nrow(f), sample.size))
  } else {
    samples <- combn(1:nrow(f), sample.size)
  x <- apply(samples, 2, function(i) get.coef(f[i, ]))
# Compare these statistics, using their variation across all possible 
# samples to establish relative scales, to their values for the data
# and select the closest.  (One might retain the several best, rather
# than just the one best, and choose among them using additional qualitative
# criteria.)
delta <- apply((x - get.coef(f)) / apply(x, 1, sd), 2, function(y) sum(y*y))
sample.cases <- sort(samples[, which.min(delta)])
g <- f[sample.cases, ]
# Check that the best match `g` reproduces the coefficients reasonably closely.
z <- rbind(get.coef(f), get.coef(g))
rownames(z) <- c("Original", "Sample")
cat("Sample:", sample.cases)
# Plot the data and the subset to compare visually.
col.red <- "#ff000080"
plot(subset(f, select=c(temperature, avg_crawling_age)), cex=1.2)
points(subset(g, select=c(temperature, avg_crawling_age)), pch=19, col=col.red)
abline(get.coef(f), lwd=3, col="Gray")
abline(get.coef(g), lwd=3, lty=2, col=col.red)

I don't think there's a specific name for it, but it's the kind of task I've undertaken a number of times in various guises.

While the number of ways of choosing a sample of size 6 from 12 (924) is already manageable, you greatly cut down the search space by saying "of similar range".

Looking at the plot:

Crawling plot

That pretty much limits you to choosing the leftmost point from the three points at the left and the rightmost point from the two points at the right, and the four interior points from the remaining points inside the ones you choose.

If we simplify that a little and simply split the data into three subgroups of size (from left to right) 3, 7 and 2 from which we choose 1,4 and 1 point, we get a total of

$\binom{3}{1} \binom{7}{4} \binom{2}{1} = 210$, combinations which is less than a quarter the size (almost small enough to examine by hand).

Edit: Since whuber has given a comprehensive answer (one which I happily upvoted), so I won't go through the details of how to do it in this cut down case, but you could apply similar techniques to the search over a smaller space.

Working by hand

In the past when I've done it, I often do a hand search first, because it often turns out that by starting with a reasonable choice and tweaking it (swapping a few points in and out to improve the characteristics I want) I can often get something quite good enough in a few tries, saving the effort of coding a more formal search.

So for example, just by eye, I'd start with (say) these six points:

hand selected points

And then by looking at the direction of deviations of the values of the sample statistics I want (like the correlation) from the values I want them to have, and the likely effect of swapping a point in the set with a point not in, I can usually get close enough for my purposes in a minute or two.

Perhaps surprisingly, a spreadsheet like Excel is often a good tool for this task. I mark what's in (/not in) the sample with say a column of 1's and 0's and compute the statistics I want from that. A few moments of choosing which points to put in or out (swapping the 1s and 0s) and its done.

Such approaches are also useful when making up data - simulate something close to what's desired, and then manipulate a few points (adding or removing or altering) to make it look closer to what you need. Again, a spreadsheet is often a handy tool for this type of task.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.