1
$\begingroup$

I'm working with panel data. This panel dataset includes data from pupils of two kinds of schools:

  • State schools (G1) where pupils change to secondary school when they are 12 years old.

  • Private schools (G2) where pupils remain in the same school. They can follow studying in the same school when they are 12 years old.

I want to compare the change in a dependent variable between these two groups of pupils, controlling for some other independent variables.

So, is this a quasi-experimental design? Maybe G1 can be seen as a treatment group and G2 as a control although there isn't a randomized assignment? Or is this an observational study? And finally, what exactly is a correlational study? Is it quasi-experimental or observational?

$\endgroup$
1
  • $\begingroup$ Wikipedia has good definitions of quasi-experiment, natural experiment, cohort study, and observational study. Your study is an observational, longitudinal (or cohort) study. Your study doesn't look quasi-experimental to me---it would be if students were randomly assigned to G1/G2 somehow by nature. A picture of the "pyramid of evidence:" sourcesandmethods.blogspot.com/2011/05/… $\endgroup$
    – Bill
    Oct 18 '13 at 12:33
4
$\begingroup$

Personally, I don't much care for the term "quasi-experimental" but it is used a lot. The Wikipedia entry for quasi-experiment seems to be good. Another way to think about it is that in a true experiment there is random selection and random assignment, but in an observational study there is neither. In a quasi-experiment there is one or the other but not both.

In your particular case, you seem to have no control over either selection or assignment, so I would call it an observational study.

As for "correlational" I've seen this used by many of my doctoral student clients. I think its frequent use comes from some book that seems to get recommended a lot. If terminology is sane, "correlational" should just mean "involving correlations", but I've seen it used for studies that involved only regressions. This terminological confusion is borne out by a Google search, which yields mostly results on sites such as "about.com". I'd avoid use of the term, myself; clearly correlations could be used in experimental designs, observational designs or pretty much any design you could come up with.

$\endgroup$
3
  • 2
    $\begingroup$ +1 I have also frequently seen (and perhaps used) “correlational” as a synonym for “observational” or “non-experimental”. At this stage, I don't think there is much hope for another, more specific definition to take hold. $\endgroup$
    – Gala
    Oct 18 '13 at 11:21
  • 2
    $\begingroup$ That is a very strict definition of "true experiment." The populations in medical trials are generally convenience samples, for example, who are then randomly assigned. So, they would be quasi-experiments. Most people seem to mean just "random assignment" by experimental. $\endgroup$
    – Bill
    Oct 18 '13 at 12:31
  • $\begingroup$ @Bill Indeed. Both "quasi-experiment" and "experiment" are used differently by different people. But this masks problems of external validity with convenience samples. $\endgroup$
    – Peter Flom
    Oct 18 '13 at 13:21
3
$\begingroup$

The use of terminology differs across disciplines. I'm answering as a political scientist.

Your study would only be a (quasi-)experimental design if there were some aspect of randomization in assignment to treatment. Based on the information you've provided, you are dealing with an observational study because the units self-assigned to treatment. This means that there are measurable and unmeasurable features of the observations that differ systematically between units in G1 and G2. In a school setting, there are essential features of students who go to state schools and those who go to private schools. Ideally (but probably not practically) you could control for these differences. I am skeptical because there are probably unmeasurable differences between the populations, such as motivation, family support and upbringing. These are things you cannot control for.

The key feature of any experiment is random assignment to treatment. If units are selected at random to receive treatment and then all units adhere to their assignment, then in expectation all treatment units and control units will be similar in their pre-treatment characteristics. Then the researcher can credibly infer that any post-treatment differences are due to the effect of treatment.

An "experiment" involves an intervention by a researcher that randomly assigns treatment. In my line of work, random selection is only possible in a very limited range of research designs. As a result, I do not consider it a requirement for a "good" or "true" experiment in my field.

A "natural experiment" involves randomized assignment to treatment through an intervention executed by something (human or not) other than the researcher.

In my experience, "quasi-experiment" refers to a research design where there is some aspect of randomness to assignment to treatment but not to a degree where the researcher believes that treatment and control units are similar in expectation in terms of pre-treatment features. This term, however, is quite vague and subject to different meanings. In my field people advise against its use. I've heard some claim it's too vague to be useful, obscuring more than it reveals about the research design. I've also heard some claim that "quasi-experiment" is just a euphemism for "bad experiment." I agree with the former sentiment and tend to disagree with the latter.

An "observational study" involves no randomized assignment to treatment. In these settings a researcher must try to establish "conditional ignorability." The term means that after controlling for a set of covariates $X$, a unit's potential outcomes are conditionally independent from assignment to treatment. In other words, the treatment assignment and the potential outcomes are conditionally independent given $X$ if and only if, given knowledge that $X=x$, knowledge of whether $T=\{0,1\}$ provides no information on the likelihood of a particular outcome in $Y$.

That is, conditional on the covariates, the treatment is independent of the potential outcomes.

\begin{equation} (Y(T=1),Y(T=0))⊥T|X \end{equation}

A "correlational study" would probably not be of much value. A "correlational analysis," however, probably means that you present exploratory data analysis that shows the unadjusted relationship between variables. In other words, without any claim of causality, you are showing how variables tend to move together in your data set.

The formalization above is based on the Rubin potential outcomes framework for causal inference, a foundational work for this conversation. You need to read it if you're serious about moving forward with work like this. Holland, Paul W. 1986. “Statistics and Causal Inference.” Journal of the American Statistical Association. 81(396): 945-960.

Another approach to conditional ignorability is Judea Pearl's back-door criterion.

$\endgroup$
1
  • 1
    $\begingroup$ Wish I had more ability to upvote. One feature not discussed is "blinding" and the need to assess the success of randomization and stability of results under resampling. $\endgroup$
    – DWin
    Oct 19 '13 at 18:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.