Is there a "best" way to select a subset of data (observations, not variables) to use in a multiple regression?

The problem: A colleague of mine is planning a project for this field season. There are a fixed number (say 200) of structures on inventory, each with associated site-level data that is already known. The intent is to visit a subset of these sites (say 100) this summer, and record response variable(s) of interest. I don't know yet what analysis will be conducted, but let's say for discussion that it will be some form of multiple regression, with the existing site-level data as a suite of predictor variables, and the data collected this summer as a response. Budget and time constraints make it impossible to sample all 200 sites, but we want our subset of sites to retain as much information as possible.

We could randomly select 100 of the 200 possible sites and trust randomness to result in a representative sample, but is there a better way, using the information that we have? Intuitively, I would expect that we would want to select sites that are as different from one another as possible.

For an initial exploration, I ran a quick simulation, simplifying the problem to a response variable $y$ and two explanatory variables $x_1$ and $x_2$. For each of 10,000 iterations, I generated a dataset and performed the following regressions, and stored the coefficient estimates and standard errors (I can provide R code for this you'd like, but this may add unnecessary clutter to my question):

  1. All data ($n=200$), for comparison
  2. Simple random sample of data, of size $n=100$
  3. Idea: maximize leverage:
    • standardize $x_1$ and $x_2$ according to the form $x_{1,c}=\frac{x_1-mean(x_1)}{sd(x_1)}$ and $x_{2,c}=\frac{x_2-mean(x_2)}{sd(x_2)}$
    • calculate Euclidean-ish distance between the standardized variables and center, according to the form $d=\sqrt{x_{1,c}^2 + x_{2,c}^2}$
    • select the $n=100$ data points farthest from the center
  4. Idea: maximize distance between data points and one another
    • standardize variables as above
    • calculate a Euclidean distance matrix
    • select the $n=100$ data points for which the sum of distances to all other points is greatest
  5. Idea: reject points that are too similar to others
    • Using the Euclidean distance matrix of standardized variables, iteratively identify the pair of data points that is closest to one another and reject one, until $n=100$ remain
  6. Idea: agglomerative clustering in reverse
    • Use R's hclust and cutree functions to hierarchically cluster the data into $n=100$ groups
    • Randomly select one data point from each.

SO WHAT HAPPENED? Here are my observations so far from the simulation:

  • All methods seemed to result in unbiased parameter estimates. In terms of central trend, there was strong agreement between the coefficients estimated by the subset regressions (2)-(6) as compared to the regression using the full dataset (1). No big surprise, but good to see.
  • There was also good agreement between the (empirical) mean SE's and the (empirical) standard deviations of the coefficient estimates themselves. Perhaps this is a minor point, but tells me that reported coefficient SE's are fairly reliable.
  • The simple random sample (2) resulted in the largest variances of the coefficient estimates, while the other methods (3)-(6) resulted in smaller variances, approaching those from the full data (1)!
  • When $x_1$ and $x_2$ were generated independently, methods (3) and (4) outperformed (5) and (6) in terms of variance reduction, with (5) slightly outperforming (6). Order in increasing variance: 1, 3/4 tie, 5, 6, 2.
  • When collinearity was introduced between $x_1$ and $x_2$, (5) and (6) outperformed (3) and (4)! (5) outperformed (6) again, and (4) outperformed (3). Order in increasing variance: 1, 5, 6, 4, 3, 2.

While this was a fun exercise, my proposed solutions felt increasingly ad-hoc. Left to my own devices, I would be tempted to pick (5). BUT ... Is there a better, more defensible way to approach site selection for this study?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.