Textbooks/readings on what to do when you can't create an ideal experiment?

Question

My statistical training is rooted in mathematical statistics, and taking these methods classes in my M.S. are a bit of a shock at the moment; it is currently difficult for me to be able to understand some of these "applied" methods since I lack experience in the industry.

One of the topics that we've been talking about in my methods classes is the idea of experimental design.

Say, for example, I want to perform an experiment on the effectiveness on an educational program that claims to raise test scores of K-12 students.

In the methods classes, they've taught the following to pursue such a problem: make sure you have a good research question, a good data gathering method, a randomized experiment, homogeneous treatment groups (i.e., one treated with this program, one perhaps not) ideally of equal size, and then run a $t$-test (or some sort of nonparametric hypothesis test), and it's all fine and dandy, right?

I have little faith that this is how it works in reality.

I've learned that, sure, you might have to do some convenience sampling. But other than that, I have no idea how to implement experimental design other than what I've learned from a textbook.

Are there any textbooks, readings, etc. that explore these issues in practice (and ideally, don't gloss over the math - I don't need detailed proofs of everything, but I don't want to be told that everything is "obvious," for example)?

Answer 1

There are two fields where randomized experiments are almost always impossible: they are social sciences and economics. In these instances you can only do "quasi experiments". Try searching with keywords quasi experiments, observational studies and social sciences; you will get some good text books. I can recommend two excellent books on this subject: the second book by Shadish and Cook is a classic:

1. Counterfactuals and Causal Inference: Methods and Principles for Social Research By Morgan and Winship
2. Experimental and Quasi-Experimental Designs for Generalized Causal Inference by by William R. Shadish and Thomas D. Cook

A classic paper that uses a technique called "propensity score matching" in non experimental setting for causal inference by Dehejia and Wahba is highly recommended as well.

Additional recommendations:

1. Design of Observational Studies by Paul R. Rosenbaum.
2. Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction by Imbens and Rubin.

IF you are looking at time series quasi experiments, the above books have some chapters devoted to them, but a dedicated book is by Gene v. Glass Design and Analysis of Time-Series Experiments and I would check his article Interrupted time series.

Trivia: Gene V Glass coined the term "Meta Analysis".

Answer 2

This is where quasiexperimental designs can be useful. In many situations in practice, experimental designs are not practical because, although you have a treatment, you are not able to perform random assignment to groups or maybe you only have one group.

In your education example, you may not have control over who receives the treatment because you intend to perform the intervention to all the kids in one school. However, you maybe able to compare their scores to scores from previous years, or randomize classrooms so that some classes receive the intervention before others, or compare multiple schools including those that did not receive the intervention.

It might make sense to do an interrupted time series design where you have just one group, but take measurements constantly, and administer the treatment in the middle of your study duration. This way, you can see if the slope of the dependent variable over time changed right after the treatment, relative to the overall slope across the entire study. The number of measurements can be as low as 3, but more the better.

So, my suggestion is to read up on quasiexperimental study designs.

Answer 3

The most thorough, general, and precise treatment of causality is Judea Pearl 2009, "Causality", 2nd ed., Cambridge University Press.

Especially, it makes clear that causality is not really a statistical issue - even unlimited data does not solve it. It introduces a precise language to express qualitative and theoretical knowledge needed for causal inference when something about the data is suboptimal. You will see that failed randomization is just one issue among many. It also subsume all other mathematical frameworks, e.g. those by Imbens, Rubin, and Rosenbaum. I can not overstate how accessible, elegant and powerful his approach is.

I strongly recommend it. However, you should read it in a non-linear fashion (chapters 5 and 11 are more accessible, and then you can work backwards through chapters 1, 3, and 7 for understanding the general theory).

When you have understand the basics, you can easily look into more recent advancements, for example on when it is possible to "transport" causal findings from one context to another, which is not necessarily possible even with randomization (Pearl, Judea, and Elias Bareinboim 2014, "External validity: From do-calculus to transportability across populations." Statistical Science).

Answer 4

Perhaps these are what you're looking for...

Statistics for Experimenters

Design and Analysis of Experiments

Design and Analysis of Experiments with R (not related to the previous title)

Process Improvement using Data (free online or as PDF, chapter 5 covers DoE)