What if a non-random sample is identical to a random sample?

Question

Sometimes, in political polls, pollsters take non-random samples from a given population, but then they apply the results of the theory of random sampling to their non-random sample. I've heard someone (not a statistician) argue that this is still valid procedure because the non-random sample obtained is one of the possible random samples.

In fact, suppose the following happens: Researcher 1, through some non-random sampling method, selects individuals A, B, C. Researcher 2 makes use of random sampling, and obtains the same sample A, B, C. Both apply random sampling theory to analyse their sample. What's the difference? What makes researcher 1 wrong?

Thoughts

My only thoughts abouts this, at least so far, is that what makes the random sample theoretically valid is the procedure that random sampling dictates, and not the particular sample obtained.

If that wasn't the case, you could fix basically any sample you want (say, a sample of 3000 white, 24-year-old, college-educated women), then claim that this sample is okay to use because it is one of the possible random samples of 3000 people of your population.

Answer 1

A particularly biased / non-representative sample is unlikely if you sample randomly.

In an ideal world you'd have a non-random sample which perfectly accurately represents the population such that the proportion of every demographic is the same in the sample as it is in the population as a whole.

This is pretty hard problem to solve in the real world though (to say the least), as you'd need to understand every demographic and how it affects your results. You might say "white, 24-year-old, college-educated women" is specific enough and you just need to make sure your sample has the right proportion of such people (and similarly for every other similar demographic), but they may be more or less likely to act in a certain way based on where they live, where they studied, where they grew up, their religion and many other factors. So you need to take all of that into account too. That'll be a whole lot of work, and in the process you'll probably answer your original query anyway without ever using the sample you generated. Basically doing that just doesn't make a whole lot of sense.

In the real world a random sample is a "good enough" attempt to obtain an accurate representation of the population.

Now it is indeed possible to get a random sample that doesn't reflect what the population as a whole looks like particularly well (i.e. a "biased" sample).

But the probability of getting any given sample when sampling randomly decreases significantly as the sample becomes more biased and a less accurate representation of the population as a whole. This applies especially when you have larger samples.

This is acceptable since statistics is generally about having high confidence of being correct rather than having absolute certainty.

Think of it this way: if 70% of your population is women and you randomly pick one person, you have a 70% chance of picking a woman. So you would expect roughly 70% of your random sample to be women. The maths might not work out to exactly 70% in all cases, but that's the general idea. So the sample proportions should roughly correspond to the proportions of the population as a whole. You should be rather surprised if your sample somehow ends up with 0% women.

---

There could also be issues depending on how you obtain a random sample. If you want to sample from everyone living in a country, you could, for example, get a random subset of registered voters or people with driver's licences. But then your sample would be heavily biased towards people who are registered to vote or have driver's licences.

This may also lead to a partially random sample where you combine differently-sized random samples from different sources such that the end result is more representative of the population as a whole. Although I'm not sure whether and how often this is done in practice. Finding a single data source for the entire population would be preferable.

But that's a whole other question.

Answer 2

Play poker with your friend, bet a lot of money, and cheat to give yourself a royal flush (it beats every other hand).

“That’s cheating!”

“Nah, it’s one of the possible hands. Pay up.”

Yes, it’s about the procedure.

(Don’t actually do the poker trick, but I think it makes the point.)

Answer 3

The central issue that has not been explicitly addressed, is that when sampling is correctly performed (randomness being one criterion), the resulting sample is a faithful representation of the underlying distribution of the population being sampled. This is what allows us to make a meaningful inference about the population from the sample.

When a sample is not chosen at random, depending on how it is chosen, any resulting inference is distorted because the sample is no longer necessarily representative of the likelihoods of the outcomes that were observed.

It is important to phrase it this way because non-random sampling does not imply that rare or unlikely outcomes are overly represented. You could, for instance, always select the mode of a binomial random variable--this is clearly not random. And it still violates the notion that the sample represents the population.

Answer 4

This illustrates the unidirectionality of conditional probabilities. Given a particular a sample and a hypothesis with well-defined probabilities, we can say with confidence what the probability, given the hypothesis, of seeing the sample. But in frequentist statistics, we cannot say what the probability, given the sample, of the hypothesis is.

That the sample is taken randomly is usually not explicitly stated as part of the null hypothesis, but it is always implicitly part of it. When we reject the null, we reject all of the null. And remember that the negation of a statement with "and" turns into a statement with "or". So if the null is "the sample is drawn from a distribution that is normal and the mean is $\mu$ and the standard deviation is $\sigma$ and the samples are independent of each other, and ..." then rejecting the null means that we believe that ""the sample is not drawn from a distribution that is normal or the mean is not $\mu$ or the standard deviation is not $\sigma$ or the samples are not independent of each other, or ..." It's only by eliminating the possibility that the sample was cherry picked that we can definitively conclude that one of the other possibilities holds.

For a Bayesian perspective, this shows the importance of updating not only on your knowledge but also on your meta-knowledge. That is, not only what you know, but how you know it. Much of the controversy surrounding the Monty Hall problem comes from the ambiguous nature of the metaknowledge. If the host always randomly picks from the two unchosen doors and shows what's behind it, then switching doesn't help your odds. But if the host always picks a door with a goat and opens it, then switching does help your odds.

Another puzzle is "Suppose you know a particular woman has two children, and you know that one of her children is a boy. What's the probability that she has two boys?" The answer depends on how you know that one of her children is a boy. If you asked whether her older child is a boy, and she said yes, then the probability is 1/2. But if you asked her whether any of her children are boys, and she said yes, then the probability is 1/3.

Answer 5

Sometimes, in political polls, pollsters take non-random samples from a given population,

This is a bit ambiguous. Very often samples are not completely randomised and there are some selection biases. But still, the results from this non-random selection might be in some way random.

The question is by how much the selection effect and the related bias is negligible.

A poll among your close friends is not a good representation. Neither is a poll on some website. However a polling organisation that selects a representative mixture of the population is probably gonna get close to the true answer.

The selection by the polling agency might be random or not, that doesn't really matter.

Urn example

Say there are 100 urns labeled $i,j$ with $1\leq i\leq25$ and $1\leq j \leq 4$.

The urns contain blue and red balls with fractions that are determined by a random process. The random process is likely depending on $j$ but not so much on $i$.

We want to know the fraction of red and blue balls in the total of all urns.

Say that we can only sample twelve of those urns due to limitations of resources. We can randomize our samples in different ways:

• We could make a random selection out of the 100 urns, but we could also decide to fix our pick (non randomly) to 3 urns out of each of the 4 $j$ categories.
• We could randomly select 3 $i$ out of each $j$ but we could also select some specific $i$ (because it might be more convenient).

All these non-random choices introduce potential bias. But that bias might be negligible if we consider that the intentional choices have only a small effect on bias.

Also note that in the end the sampling process is still giving a random variable (but only biased random). We might have selected some urn labels $i$ non randomly, but how the balls got inside the urns is still a random process, a random value.

The issue with non randomised sampling methods is not that the outcome variable is not random, but that the outcome variable might be biased.

E.g. that poll among your friends is still a random variable.