Factorial experiments have several advantages when one wants to examine the effect of several independent variables (factors) on an outcome. Perhaps most importantly, they often allow additional factors without major increases in sample size.

However, sampling is often the most difficult thing to begin with, and in some cases, even a 2x2 factorial design can require a sample that is practically impossible to obtain. It is not necessarily the sample size that is the problem, but the feasibility of obtaining a random sample.

I'm considering a case where, for practical and ethical reasons, I have no options but to compare a preselected group of people to another group which I can select randomly from a larger population. The purpose is to examine whether a certain treatment has an effect on a behaviour which everyone in the population (including the preselected group) will carry out continuously in collaboration with each other. Everyone in the population has gone through an integration process in which they have learned, one way or the other, the behaviour. The question is whether the treatment is effective for quickly learning the desired behaviour in the preselected group – is the behaviour learned in a shorter time and with more accuracy than what previous newcomers have been able to learn without the treatment. There will also be other factors in the design except the treatment.

In an ideal case, I could randomly select two groups that are part of the population, and give the treatment to one of them. However, it is impossible in this case, because I cannot give the treatment to anyone but the preselected group. In addition, due to ethical reasons, I cannot randomly split the preselected group in half and give the treatment only to one half: everyone in the group must get the treatment. Thus I am forced to give the treatment to everyone in the preselected group and use a random sample of the existing population as a proxy for the non-treatment group.

Are there any valid ways of conducting a factorial experiment under these conditions where I have to deal with a non-random sample? If not, are there any other ways to assess the effect of the treatment under these conditions?


If you don't have a random group of subjects, I don't think you can say you have an experimental design. You could call it quasi-experimental, perhaps, of just observational.

You can do ANOVAs on observational studies - that's fine. You just have to be careful about generalization - who are you inferring to?

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.