Quasi and True Experiments

Question

If subjects are recruited using a convenience sample to take part in a study, and each participant is then randomly assigned to either treatment condition, A and B.

(i) Will my result be generalizable to the broader population? Yes or No

(ii) We need to be realistic that random sampling is not always possible, but random assignment to a treatment condition is possible. Statistically speaking, what is the advantage of performing an experiment such as that described above?

(iii) If we are studying the relationship between two or more variables in this convenience sample that had random assignment, can this be inferred to the larger population?

Answer 1

The answer to your question depends strongly on features of your experiment that aren't detailed in your description.

Consider a study that examines the limit of light detection. It turns out that the variability and absolute limit are very similar across people with otherwise good vision. A convenience sample is often perfectly adequate in this case and likely generalizes to the population of people with otherwise similar vision. However, if I were wanting to rank the production resources I put into various colours of my widget I wouldn't just ask the executives of the company to participate in a survey.

Answer 2

Short answer: Random assignment is a requirement for an experiment. It's necessary, but not sufficient.

Your question has a strong word "convenience sample" which triggers all kinds of red flags. However, you may not mean it that way.

It's obviously possible to recruit participants to a study in a scientifically rigorous manner. Medical studies do this all the time. But so do pop-psychology, recruit-some-college-students studies that are pure junk. Which are we talking about here? For the former, random assignment is a link in a solid chain, for the later it's a bandaid that's attempting to make things look scientific.

Once you've gotten a haphazard sample of convenience, nothing can get you out of that box. Nothing. So if you mean "can I recruit some students from my college for a psychology test and after that pretend it's a scientific experiment on the world's population" the answer is No. No for (i), No for (ii), No for (iii). (Again, I'd argue that (i) and (iii) are the same question, with no meaningful distinction.)

On the other hand, you could spend a LOT of time, planning, and effort to recruit a sample from the population you intend to generalize to. Or you can be satisfied with only drawing conclusions for people like the sample you actually got. Recruiting is a complication, but it's not the problem.

A lot of work outside of a laboratory is done with found data rather than experimental data. The line between "found data" and "convenience sample" is not well-defined -- certainly it's much less clear than the line between experimental data and the other two. With found data, the question is also how generalizable the results are, and there are actually Found Experiments which were unintended.

This is basic logic: a logical argument is as weak as its weakest link, and each link builds on prior links. Random assignment is necessary for a generalizable experiment, but not sufficient. Again, my (original, now deleted) example: you recruit some brothers from your fraternity to test the efficacy of (female) birth control pills. You can be as rigorous as you want after that -- double-blind, random assignment, independent verification, replication, etc -- but your initial sample put you in a box you can't get out of. You'll never be able to generalize the results to women.