Deviation from sampling plan

Question

I have data coming from a sample that, for various reasons, don't follow the original sampling plan. Trying to calibrate this sample seems to be very difficult because deviations are hard in some cases.

At first the design aimed to include a number of small regions but not all of them have participated in the study. If we consider bigger areas we have data from all of them but the distribution in the sample is very different from the population (even for post-calibration techniques).

As we have a large amount of data, would it be a good solution to draw a random subsample from this dataset following the probabilities observed in the population by different characteristics?

Answer 1

There are situations in survey analysis when you do a random resample similar to what you describe, with the intent of creating a new sample that can be treated as though it is a simple random sample. For example, this is sometimes done so the new sample can be plotted with each point having equal weight.

However, for general purposes this is less useful than keeping the full sample that you have gathered and using post-stratification weighting on it. If you know enough about the probabilities to resample in accordance with the population distribution, you know enough to give weights to the points in your original sample. This retains more of the information in your original sample but allows the overrepresented regions to be downweighted accordingly.

Basically, what I am suggesting is that though what you suggest is a possible solution, if you can implement it you can also implement a traditional weighting scheme, and retain your full sample. You will then need to analyse it using the usual techniques for complex surveys rather than for simple random sampling; but you will have a better solution than your resampling approach.

I recommend Thomas Lumley Complex Surveys: a guide to analysis using R published by Wiley, even if you aren't using R.