4
$\begingroup$

Introduction I am confused about when a sample is a random sample (i.e. probability sample) and when it is a purposive sample (i.e. non-probability sample). My understanding is that the former allows me to conduct statistical analyses and generalize to a larger population but that the latter precludes the ability to estimate a sampling error and, therefore, precludes the ability to make statistical inferences about a population.

Scenario 1 As an example, consider a regulatory agency auditing pet stores in Arizona, trying to determine whether a pet store is up-to-date on vaccinations for its animals. Thus, the inclusion criteria in the sampling frame consists of all pet stores in Arizona and the regulatory agency can randomly sample from that list and it would be a random sample, allowing them to generalize the findings to all pet stores in Arizona, correct?

Scenario 2 However, to save on costs, let's say they impose some exclusion criteria to limit the universe by removing those pet stores that are "certified" (i.e. they have passed a training program on animal vaccination), and therefore unlikely to be malfeasant. If the regulatory excludes them, the sampling frame will become "all non-certified pet stores in Arizona". If a random sample is then drawn from this limited sampling frame, is it still a random sample? I think so, with the caveat that I cannot generalize the findings to all pet stores in Arizona, but only to "all non-certified pet stores in Arizona", right?

Scenario 3 Finally, let's say that instead of exclusion criteria based on a clear-cut subgroup such as "certified" stores, what if an expert in the field curated the sampling frame, crossing off the list those stores deemed to be "unlikely offenders", based on his knowledge of the pet store industry. Once the list is curated, the remaining pet stores will be randomly selected, but this no longer seems like a random sample. Instead, it seems like it's a purposive sample, or perhaps a judgment sample, correct? If so, what are the implications for statistically analyzing the findings?

$\endgroup$
4
$\begingroup$

Edit: Each scenario has a sampling frame, and a random sample was taken from that frame. The frames differ, and thus the parts of the population sampled differ. It is inaccurate to describe the design in any of the scenarios as a judgement or purposive sample.

Scenario 1: the frame is the entire population, so the sample does generalize to the entire population. I agree with your evaluation.

Scenario 2: The frame is an objectively defined subset of the population. In other words, anyone with access to the proper records would include the same stores in the frame; there's no judgement involved. The sample generalizes to the population of not-certified stores. What you can say about the larger population depends on the study outcomes. Suppose the primary outcome is a proportion, which is expected to be higher for not-certified stores. There are many instances where one would have either prior data or expert opinion would provide an upper bound for the stores that were not studied. Let it be, for argument's sake, 5%. Then one can estimate the proportion in the entire population as follows. Let there be $N$ stores in the population, $N_1$ not certified and and $N_2$ certified. Let the sample proportion with the outcome in the non-certified stores be $p_1$. Then an upper bound for point estimate in the whole population is:

$$ p \le \frac{N_1}{N} p_1 + \frac{N_2}{N} 0.05 $$ Similarly, if $p_1^L$ and $p_1^U$ are the lower and upper confidence $1- \alpha$ limits for population proportion in the not-certified stores, then one has an upper bound that takes into account sampling variablity:

$$ p \le \frac{N_1}{N} p_1^U + \frac{N_2}{N} 0.05 $$

In some instances, expert opinion might also give a lower bound for the excluded part of the population, say 0.01 (1%), giving the interval:

$$ \frac{N_1}{N} p_1^L -\frac{N_2}{N} 0.01 \le p \le \frac{N_1}{N} p_1^U + \frac{N_2}{N} 0.05 $$ This kind of design is actually quite common, even good practice. So the sampled population is well-defined; and one can say something about the entire population.

Scenario 3 is interesting. It is actually identical in structure to Scenario 2. It is not a judgement or purposive sample. By your description, the sample is entirely random, with selection determined by random numbers only. The expert judgement went into the definition of the frame: he individually classified all stores. In this kind of work, there's no such thing as a perfect frame. So for a survey in one time and place (like this) this would be an acceptable design. I'd really want to satisfy myself as to the reasoning behind the expert's choices.

The real problems would come if one wanted to replicate the study at other times and places, without the original expert.

I would expect to see an analysis similar to that described for Scenario 2. For generalizing to the population, analysts might be more uncertain about the bounds for estimates about the excluded parts, because there is likely to be no prior data about an ill-defined population.

Added: I actually worked on a study in which an expert classified units for the frame. The intention was to analyze the effect of lead paint in children. Therefore it was necessary to classify individual dwellings in a neighborhood as likely to have lead paint or not. The expert was a professor of industrial hygiene, who judged on the basis of external appearance. He was able to describe the criteria he used. He knew, for example, that subsidized housing would not have lead paint.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.