Randomisation of non-random sample

Question

I am always a bit surprised to see psychological adverts for participating in experimental studies. For sure, people who respond to these adverts are not randomly sampled and therefore are a self-selected population.

Because is it known that randomisation solve the self-selection problem, I was wondering if randomisation of a non-random sample actually changed anything.

What do you think ? And also, what should we make of all these psychological experiments based on heavily self-selected sample ?

Answer 1

Randomization in a non-random sample can still show an effect is not reasonably explained by random variation.

For example imagine we have a population with two unrecognized subgroups (with somewhat different characteristics*) of roughly equal size, but your sample is non-random, giving an 80/20 split. Let's imagine 2 treatment groups of equal size. Randomization (at least with decent sample sizes) will tend to give close to that 80/20 split in each group, so that treatment effects are due to the treatment, rather than unequal allocation of the heterogeneous groups to the treatments.

* leading to different baseline means, say

The problem comes when you want to extend the inference to some target population other than what your sample is representative of (the self-selectors); this requires assumptions/an argument for which you may have no evidence (such as assuming that say the treatment differences will be consistent for all subsets of the population).

For a similar situation, imagine testing a hypertension drug only on men, compared to a standard treatment and placebo. Assume the men are properly randomized to treatment group. A treatment effect will be real in the sense that it really does describe an effect in men. The difficulty will come when trying to extend that inference to women.

So if they're properly conducted and randomized apart from the recruitment, an observed significant effect will be what it seems, but it will apply to what you actually sampled, not necessarily what your desired target was -- crossing the gap between the two may require careful argument; such argument is often absent.

When I was a student it was quite common for psychology experiments to be conducted on psychology students, who were expected to volunteer for a certain number of hours of such experiments (this may still be the case but I don't have regular contact with psychologists who do experiments any more). With randomization to treatment, the inferences may have been valid (depending on what was done) but would apply to the local population of self-selected psychology undergraduates (in that they generally choose which experiments to sign up for), who are very far from a random sample of the broader population.

Answer 2

Because is it known that randomisation solve the self-selection problem, I was wondering if randomisation of a non-random sample actually changed anything.

In short, no. Think of it this way: you have an urn with 100 black balls and 100 white balls. You sample 90 black balls and 10 white balls from it. Sampling randomly from this subsample will not allow you unbiased inference on the urn itself.

And also, what should we make of all these psychological experiments based on heavily self-selected sample ?

People agree that non-random sampling is a problem. But how much of a problem is also a question of your "theory" of the mechanism you're interested in. If your hypothesis deals with a mechanism that should basically be the same for all humans (i.e. experiencing a freezing sensation when dipped in icy water), then non-random selection doesn't matter that much. Unfortunately, that's often not the things we're interested in.

Answer 3

There is a technique designed to deal with the issues you mention known as Bootstrapping. Bootstrapping is an approach where you generate new synthetic samples by drawing from your actual sample pool with replacement. You then do statistics on each of those synthetic sample pools, and compare the statistics between sets.

This has a strong advantage of allowing you to use a great many additional tools in your statistics because these synthetic samples come from a known distribution. You can then determine how good your estimators are at handling these synthetic cases. If you find that the estimators for all of your synthetic samples converge nicely onto the same result, the assumptions of bootstrapping allow you to infer that your estimators, when applied to the full sample, provide good estimates for the unknown population. If, on the other hand, you find your estimators yield very different results from synthetic sample set to synthetic sample set, you should infer that your estimators, when applied to the full sample, may not provide a very good estimate for the unknown population.

This bootstrapping approach can be used to validate whether the randomization of your non-random sample is sufficient. It can't prove it, of course, but it has been used as a tool to enhance credibility by double checking your assumption that your random sampling is sufficiently random.