comparing two samples drawn using two different sampling methods

This is a hypothetical question, so I don't have a lot of additional details to give. However my question is pretty straightforward:

Is it theoretically valid to conduct tests (e.g. for comparing means or percentages), between two samples that have been drawn using different sampling methods?

For instance, what about a comparison of means (t-test) between a sample drawn using stratified sampling, and a sample drawn with simple random sampling? (these are just two examples, my question is about any sampling method that on its own allows statistical inference).

I'm thinking about a situation where we would want to compare two different populations that have been sampled differently.

Assuming this kind of comparison is possible, I guess it requires some particular process and conditions. What are they? How should I proceed? Thanks.

• I don't see why not. As a thought experiment, consider Welch's t-test. The test statistic is $(\bar x_1 - \bar x_2) / \sqrt{s^2_{\bar x_1} + s^2_{\bar x_2}}$. All of these quantities are either sample means and sample standard errors. These can be calculated separately, using survey-design procedures. I'm not sure what the degrees of freedom would be off the top of my head, but a decent first cut would be the number of primary sampling units. Commented Dec 13, 2023 at 21:59

The question is whether you can estimate the variances and covariance of the means (and the biases, potentially). For usual survey settings you will be able to estimate the variance of each mean, and they should be approximately unbiased, but they might not be independent.

For example, suppose your two surveys were both multistage cluster samples conducted by the US National Center for Health Statistics. It's quite likely that the choice of primary sampling units for the two surveys will be correlated -- for example, both surveys are likely to include the largest urban areas with probability 1. They are likely to be uncorrelated at the final stages of sampling individuals. It might be hard to find out whether the two estimates can sensibly be treated as independent. Or if the two surveys were both business surveys of some sort, they might well oversample large business in similar ways and end up correlated.

If you're doing secondary analyses of public-use data, you probably won't have the information needed to estimate the covariance, so you will probably need help from the people who conducted the survey. Given knowledge of the sampling schemes in the survey the procedure is fairly straightforward

• if the two estimates can be treated as independent, then just estimate the means and variances and do a t-test directly
• if they can't be treated as independent then treat them as a single dual-frame survey sample

You originally asked about respondent-directed sampling. This makes the whole problem more difficult, because there aren't very good estimates even of the bias and variance for RDS, so it would be hard to be confident about the test results. RDS (to the extent I know about it) is most useful when getting a sufficiently-large, qualitatively representative sample is difficult and you're willing to compromise on the extent to which it's really a probability sample. I think it would be challenging to find reliable estimates of covariance between an RDS sample and anything else.

• +1. Your discussion of how separate hierarchical samples can fail to be independent is instructive.
– whuber
Commented Dec 13, 2023 at 23:18
• thanks! I initially removed the reference to RDS from my question in order to simplify it, and make it more likely to get answers, but thanks for addressing it, I appreciate that. Commented Dec 14, 2023 at 3:14