What examples of lurking variables in controlled experiments are there in publications?

Question

In this paper:

Lurking Variables: Some Examples Brian L. Joiner The American Statistician Vol. 35, No. 4, Nov., 1981 227-233

Brian Joiner claims that "randomization is not a panacea". This is contrary to common statements such as the one below:

A well-designed experiment includes design features that allow researchers to eliminate extraneous variables as an explanation for the observed relationship between the independent variable(s) and the dependent variable. These extraneous variables are called lurking variables.

The quote was taken from this question and does not have a source but in my experience it is representative of the prevailing attitude: Examples of Lurking Variable and Influential Observation

One example given is that when testing the safety (specifically carcinogenesis) of red #40 food dye on rodents in the seventies an effect of cage position was found to confound the study. Now I have read many journal articles studying carcinogenesis in rodents and have never seen anyone report controlling for this effect.

Further discussion of these studies can be found here: A case study of statistics in the regulatory process: the FD&C Red No. 40 experiments.

I could not find a non-paywalled version but here is an excerpt:

At the January meeting, we presented a preliminary analysis (14) that disclosed a strong correlation between cage row and RE (reticulo-endothelial tumor) death rates, which varied from 17% (bottom row) to 32% (top row) (table 2). We could not explain this strong association by sex, dosage group, or rack column or position. A subsequent analysis (18) also indicated that cage position (front vs. back) might be correlated with non-RE mortality and that position was correlated with time to non-RE death.

I am specifically interested in why there seems to be such a problem with replication in the medical literature, but examples from all fields would be welcome. Note that I am interested in examples from randomized controlled experiments, not observational studies.

Answer 1

A few examples from clinical research might be variables that arise after randomization - randomization doesn't protect you from those at all. A few off the top of my head, that have been raised as either possibilities or been noted:

• Changes in behavior post voluntary adult male circumcision for the prevention of HIV
• Differential loss to follow-up between treatment and control arms of an RCT
• A more specific example might include the recent "Benefits of Universal Gowning and Gloving" study looking at prevention of hospital acquired infections (blog commentary here, the paper is behind a paywall). In addition to the intervention, and potentially because of it, both hand hygiene rates and contact rates between patients and staff/visitors changed.

Randomization protects against none of those effects, because they arise post-randomization.

Answer 2

Here is one example I found for microarray data. The measured expression has been reported to be strongly correlated with position on the "chips". This is a case where randomizing the position of the samples may lead to increased chance of making a labeling error so those doing the technical work may choose not to randomize if they do not think it is important.

Random assignment of experimental units to treatments controls the likelihood that any factor other than the treatment is the cause of the association (1,2)⁠. In some microarray platforms such as Illumina® and NimbleGenTM, multiple biological samples can be hybridized to a single chip. Chip and sample position effects may affect accuracy and reproducibility of microarray experiments unless balance and randomization is considered in the experimental design (4). Our aim was to compare the impact of these effects in a confounded and a randomized experiment.

Importance of Randomization in Microarray Experimental Designs with Illumina Platforms

Ricardo A. Verdugo, Christian F. Deschepper, and Gary A. Churchill. The Jackson Laboratory, Bar Harbor, ME 04609, Institut de Recherches Cliniques, Montreal, QC, Canada.

Answer 3

I have an example that might be somewhat different from what you originally intended when you asked this question. The past year or two have given rise to an ongoing discussion in psychology over the cause of the lack of replicability of effects from randomized experiments. Versions of this debate have surfaced for many years, but the debate has become more strident since the publication of a paper showing that many practices that are standard in psychology in the formulation of hypotheses, collection of data, analysis of data, and reporting of results allow researchers to find results supporting even arbitrarily chosen hypotheses (in the original paper, the researchers used these practices to show that listening to "When I'm Sixty-Four" by the Beatles made people younger).

The root of the problem, of course, is the pervasive incentive structures in psychology (and in other sciences) to obtain novel, positive, "publishable" results. These incentives encourage research scientists to adopt practices that, while not as obviously "wrong" as data fabrication, nonetheless lead to an increased rate of false positive results. These practices include:

1. The collection of multiple, and highly similar, dependent variables. Only the dependent variable that produces the results most consistent with the original hypothesis is reported.
2. During data collection, testing for significant results multiple times and stopping data collection when significance is obtained.
3. During analysis, the inclusion of multiple covariates in the statistical model. In the final paper, only the combination of covariates that leads to results most consistent with the original hypothesis is reported.
4. Dropping conditions that lead to results that are inconsistent with the original hyptoheses and failing to report these conditions in the paper.

And so on.

I would argue that the "lurking variable" in these cases is the incentive structure that rewards researchers for obtaining positive, "publishable" results. In fact, there have already been several high-profile results in psychology (many of which are in my specialty, social psychology) that have failed to replicate. These failures to replicate, many argue, cast doubt on entire subfields of psychology.

Of course, the problem of incentive structures that encourage false positives is not unique to psychology; this is a problem that is endemic to all of science, and thus to all randomized controlled trials.

References

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 17, 1359-1366.

Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615-631.

Yong, E. (2012). Bad copy. Nature, 485, 298-300.

Abbott, A. (2013). Disputed results a fresh blow for social psychology. Nature, 497, 16.