Observational vs quasi-experimental design?

Question

I am having some difficulty understanding the difference between and identifying an observational vs quasi-experimental design. From my understanding, an observational study is one in which the researcher does not influence the system and only records what they observe (duh). In an experimental study, the researcher manipulates the experimental units such they have different treatments and measures some resultant metric(s), usually to compare them. My understanding of quasi experimental studies is that the researcher uses groups that are already different from one another rather than apply the treatment to the EUs themselves.

For example, let's say a researcher is making observations on bird diversity in different land use types (eg forest vs agriculture). Odds are he or she isn't going to make a forest and farm and put birds in them to see which survive. They are going to go to several farms and several forests and observe the birds that live in each.

Now, it seems that this is a quasi-experimental design based on my above definitions, but does that mean that every comparison study is going to be either experimental or quasi experimental? I can't think of particularly many studies that would fall under the observational category, in that case (correlational, descriptive).

Answer 1

First, as far as you have described the research design, the study is not a quasi-experiment.

I prefer the term natural experiment to quasi-experiment, because I think it more clearly communicates the fact that treatment needs to have been randomly assigned (or as-if randomly assigned). I use the term natural experiments below, but I consider the two equivalent in meaning.

You are correct that experiments are confined to those situations where a researcher actually manipulates treatment assignment.

Observational studies comprise anything that was not an experiment. Natural experiments are a subset of observational studies, but in a natural experiment units were assigned to treatment in a random process (or as-if random, or almost random).

You might look for a natural experiment (or quasi-experiment) if you were seeking to identify the causal effect of a treatment on a set of outcomes. Then you would look for a situation where assignment to that treatment was assigned randomly (or as-if randomly) by nature or a government program, for example. For example, if you wanted to study the impact of forest fires on bird diversity, you might find a place where the government has defined that it will fight fires when they come with X miles of residential areas. After forest fires, you could compare (i) bird diversity in areas affected by the forest fire just a little further than X miles away from residential areas (treatment group) to (ii) bird diversity in areas just a little less than X miles away from residential areas (control group). Because birds would not choose where to live prior to fire based on the government's designation of the distance X, we can expect that before the fire on either side of the X-mile cutoff, birds would be identical on average. There assignment to treatment (being "treated" by the forest fire) is as-if random on either side of the X-mile cutoff. This design is called a regression-discontinuity design [1] or a geographic regression discontinuity design [2].

Also, see more discussion of the difference here: Panel study is a quasi-experimental study? Quasi-experimental is the same as correlational?

1. https://en.wikipedia.org/wiki/Regression_discontinuity_design
2. "Geographic boundaries as regression discontinuities." LJ Keele, R Titiunik. Political Analysis, 2014

Answer 2

I can try to give an example from my own field, econometrics:

Economists are interested in "returns to schooling", i.e., how much more do you earn per additional year of schooling obtained.

An experiment is not an option, as, for good and obvious reasons, you cannot force people to continue or stop their educational career just because of the empirical analysis.

Observational data is generally tricky to interpret if you are interested in the causal effect of another year of schooling, because there are "confounders" that imply that the (generally positive) correlation between schooling and earnings is not (fully) causal. For example, more able, motivated and careful individuals can be thought to choose to obtain more schooling, and such individuals would have at least partly done well in the labor market without additional schooling.

Now, sometimes nature or law is kind enough to hand you a "quasi-experiment". In the above example, researchers have for example exploited changes in compulsory schooling laws. If, effective say 1955, all students in a country are obliged to attend secondary school for, say, 10 rather than 8 years, there will be at least some students who obtain more schooling only because of the new law, and not because they choose so.

Instrumental variable approaches may then be a credible way to exploit this so-called exogenous variation to say something about the causal effect of schooling.

Answer 3

I would like to answer your question from the Epidemiology point of view.

Basically,there are three kind of studies in Epidemiology, observational study, Experimental and theoretical study.

For observational studies, as a researcher you will not give any interventions to any groups you will study. You just collect data cross-sectionally, retrospectively or prospectively.

For experimental design, as a researcher you will allocate your intervention to some groups and other groups will not receive your intervention.

There are randomized experiments (such as clinical trials) and non-randomized experiment.

For randomized experiments patient belongs to which group is determined by randomization procedures).

For non-randomized experiment, which is also called quasi-experiment, there are no randomization procedures to allocate patients to different groups, it might just be done by convenience.

Answer 4

The quasi experimental design is the one that uses an "experimental research procedure" but not all extraneous variables are controlled. Quasi experimental designs lack random assignment of participants to groups. Only in strong experimental designs is this achieved.

Causal inferences can only be done from quasi-experimental designs if (1) cause and effect covary, (2) cause must precede effect and (3) rival hypothesis must be implausible (so the relationship between variables must not be due a confounding extraneous variable).

Now, the third condition is hard to achieve since there is no randomization.

So we can see Quasi-experimental designs to be a better option than weak experimental designs and not as good as strong experimental designs.

For your bird-watching scenario, using a quasi-experimental design will not have a conclusive result and the relationship between the type of bird and the land use parameters might be affected by other variables like weather, migration seasons, temperature, humidity, wind orientation, etc. However this might be good enough for your study if you are not able to apply a strong experimental design.

On the other hand, the observational study draws inferences from a sample to a population where the independent variable is not under the control of the researcher. The observational study is then more into the data collection process, where you as a researcher must collect what you can, to draw inferences from there. The inference in this case (statistically speaking) could be managed by the amount and quality of attributes recorded. Naturalistic observation is conducted in real world observations and subject to noise and error. Since the observational study might be conducted in a single farm or a couple selected from the near-by surroundings, this will not be using a randomized sample which might also be prone to statistical error for causal relationships. The only way the observational study will be good enough to demonstrate cause and effect, will be when it is ran under laboratory conditions, say your birds are in a controlled environment, where several domes are created that represents each land/farm type and then you basically observe behavior or whatever you are measuring. The laboratory observation is closely similar to a quasi-experimental design since you are having control of a variable (the setting).

hope this helps.

Answer 5

The point of experiments is to determine causality, which typically requires establishing that: 1) one thing happened before the other, 2) that the putative cause had some explanation mechanism for affecting the outcome, and 3) that there are no competing explanations or alternate causes. Also helps if the relationship is reliable--that the lights go on every time you hit the switch. Experiments are designed to establish these relationships, by controlling conditions to establish chronological sequence and control for possible alternate causes. Effective experimental design also includes a control: A population that is not given the experimental treatment.

In many cases, it's not possible/safe/legal to establish a control group before the experiment. In which case, it's a quasi-experiment. If the assumption that the affects of the treatment were random is true (ie, people weren't somehow selected for the treatment, by age/socio-economic status/race etc), then it's a control. The assumption is random assignment, but sometimes that assumption has to be relaxed (or controlled for); if not controlled for, it weakens the strength of your causal inferences.

Control groups also rely on the assumption that the two groups prior to the treatment were identical. This measurement is typically called the a 'pre-test' measurement. Then after the experimental treatment is applied, a 'post-test measure is made. The pre-test measure should be the same for both the treatment (experimental) and control groups. And then, if the experimental treatment did anything, the post-test value for the treatment and control groups should be different. In summary, both experiments (natural and otherwise) should have a pre-test, post-test and control group.

Actual observational studies belong to a completely different style of science: inductive rather than deductive, and can generally be identified with qualitative traditions. The essential difference is that numerical data sufficient for a statistical analysis isn't available. In a comparative case study, the sample might be as small as two. While their are a variety of qualitative techniques I'm not remotely qualified to talk about, I'll make a very broad generalization and say: they compare fewer, less well defined things, because part of the role of qualitative research is to define what things are: to create constructs an definitions and theorize (based on observations) what the possible causal relationships between things might be.