3
$\begingroup$

If subjects are recruited using a convenience sample to take part in a study, and each participant is then randomly assigned to either treatment condition, A and B.

(i) Will my result be generalizable to the broader population? Yes or No

(ii) We need to be realistic that random sampling is not always possible, but random assignment to a treatment condition is possible. Statistically speaking, what is the advantage of performing an experiment such as that described above?

(iii) If we are studying the relationship between two or more variables in this convenience sample that had random assignment, can this be inferred to the larger population?

$\endgroup$
3
  • $\begingroup$ related $\endgroup$
    – Glen_b
    Dec 17, 2015 at 16:38
  • $\begingroup$ Thank you Glen for the link. However, this does not really answer my question. $\endgroup$
    – user39531
    Dec 17, 2015 at 20:45
  • $\begingroup$ Can you clarify (in an edit to your question) what your question is asking that is distinct from what's there, or explicitly identify what aspect of your question is not addressed there? $\endgroup$
    – Glen_b
    Dec 18, 2015 at 0:39

2 Answers 2

2
$\begingroup$

The answer to your question depends strongly on features of your experiment that aren't detailed in your description.

Consider a study that examines the limit of light detection. It turns out that the variability and absolute limit are very similar across people with otherwise good vision. A convenience sample is often perfectly adequate in this case and likely generalizes to the population of people with otherwise similar vision. However, if I were wanting to rank the production resources I put into various colours of my widget I wouldn't just ask the executives of the company to participate in a survey.

$\endgroup$
2
$\begingroup$

Short answer: Random assignment is a requirement for an experiment. It's necessary, but not sufficient.

Your question has a strong word "convenience sample" which triggers all kinds of red flags. However, you may not mean it that way.

It's obviously possible to recruit participants to a study in a scientifically rigorous manner. Medical studies do this all the time. But so do pop-psychology, recruit-some-college-students studies that are pure junk. Which are we talking about here? For the former, random assignment is a link in a solid chain, for the later it's a bandaid that's attempting to make things look scientific.

Once you've gotten a haphazard sample of convenience, nothing can get you out of that box. Nothing. So if you mean "can I recruit some students from my college for a psychology test and after that pretend it's a scientific experiment on the world's population" the answer is No. No for (i), No for (ii), No for (iii). (Again, I'd argue that (i) and (iii) are the same question, with no meaningful distinction.)

On the other hand, you could spend a LOT of time, planning, and effort to recruit a sample from the population you intend to generalize to. Or you can be satisfied with only drawing conclusions for people like the sample you actually got. Recruiting is a complication, but it's not the problem.

A lot of work outside of a laboratory is done with found data rather than experimental data. The line between "found data" and "convenience sample" is not well-defined -- certainly it's much less clear than the line between experimental data and the other two. With found data, the question is also how generalizable the results are, and there are actually Found Experiments which were unintended.

This is basic logic: a logical argument is as weak as its weakest link, and each link builds on prior links. Random assignment is necessary for a generalizable experiment, but not sufficient. Again, my (original, now deleted) example: you recruit some brothers from your fraternity to test the efficacy of (female) birth control pills. You can be as rigorous as you want after that -- double-blind, random assignment, independent verification, replication, etc -- but your initial sample put you in a box you can't get out of. You'll never be able to generalize the results to women.

$\endgroup$
8
  • $\begingroup$ To recap, I understand the answer to (i) is NO (ii) is acceptable in certain fields like social sciences. How about (iii)? $\endgroup$
    – user39531
    Dec 18, 2015 at 15:28
  • $\begingroup$ @user39531: Could you elaborate on the difference between "a result" and "the relationship between two or more variables"? Perhaps an example of the data you're thinking of and what kind of relationship you might find? I'm probably not being rigorous enough, but they sound the same to me. $\endgroup$
    – Wayne
    Dec 19, 2015 at 16:02
  • $\begingroup$ A "relationship" between two variables - how one independent var influences a dependent var. It can also mean how strong the relationship (r-square) between two dependent var are. $\endgroup$
    – user39531
    Dec 19, 2015 at 20:18
  • $\begingroup$ @user39531: You haven't defined "result", and I can't imagine a result that isn't a relationship (as you define it). The bottom line is not what you're trying to do, but the fact that you are trying to extrapolate outside of your defective sample. $\endgroup$
    – Wayne
    Dec 19, 2015 at 20:35
  • $\begingroup$ By "result," I mean the findings of the study based on the convenience sample. For example, if the result suggests that "what is beautiful is usable." Can this result be generalized to the broader population? $\endgroup$
    – user39531
    Dec 19, 2015 at 20:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.