Pitfalls in experimental design: Avoiding dead experiments

Question

I've come across this quote numerous times:

To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of. -- Ronald Fisher (1938)

To me, it seems perhaps a little presumptuous. The only examples I've ever found describing how experiments die without good design are around lack of controls, or poor controls. For example, experiments that control for the application of a fertilizer, but fail to control for the environment required for the application. Maybe it's just me, but it seems that a quick read through the Wikipedia section on Fisher's design principles would cover most bases.

As a statistician, how often do you see design of experiment-related problems with data? Are they always related to those few factors mentioned by Fisher, or or there other serious pitfalls that we non-statistically trained scientists should be looking out for?

Answer 1

I believe what Fisher meant in his famous quote goes beyond saying "We will do a full factorial design for our study" or another design approach. Consulting a statistician when planning the experiment means thinking about every aspect of the problem in an intelligent way, including the research objective, what variables are relevant, how to collect them, data management, pitfalls, intermediate assessment of how the experiment is going and much more. Often, I find it is important to see every aspect of the proposed experiment hand-on to really understand where the difficulties lie.

My experience is mainly from medical applications. Some of the issues I have encountered that could have been prevented by consulting a statistician beforehand:

• Insufficient sample size is, of course, number one on this list. Often, data from previous studies would have been available and it would have been easy to give a reasonable estimate of the sample size needed. In these cases, the only recourse is often to do a purely descriptive analysis of the data and promise further research in the paper (not publishing is usually not an option after doctors invested valuable time).
• Execution of the experiments is left to convenience and chance instead of design. An example I am currently working on has measurements collected over time. The measurement times, measurement frequency and end of monitoring period all vary wildly between individuals. Increasing the number of measurements per individual and fixing the measurement dates and end of monitoring period would have been fairly little extra work (in this case) and would have been very beneficial to the study.
• Poor control of nuisance factors that could have easily been controlled. E.g. measurements were sometimes performed on the day of sample collection and sometimes later, leaving the possibility that the sample has degraded.
• Poor data management, including my personal favourite "I rounded the data before putting it into the computer, because the machine is inaccurate in its measurements". Often, relevant data is just not collected and it is impossible to get it after the fact.

Often, problems with a study go even further back, to the initial conception of the research:

• Data is sometimes collected without a clear objective and just the assumption that it will be useful somehow. Producing hypotheses and "significant results" is left to the statistician.
• And the opposite: data is scraped together with the aim of proving a specific point that the PI has in his head, irrespective of the data and what can actually be proved with it. This time, the statistician is just supposed to put his stamp of significance on pre-written conclusions without the conclusions getting adjusted in the face of the data.

So far, this mainly sounds like the statistician suffers and maybe scientific integrity suffers when the PI tries to push conclusions not supported by the data (always a fun discussion). But the experimental team suffers as well, because they do unnecessary extra work (while not doing necessary work) during the experimental phase and need to spend much more time in discussion with their statistician after the fact, because they did not get their advice before. And of course, the final paper will be worse, will have fewer conclusions (and more "conjectures") and will likely not make it into that high-impact journal the PI wanted.

Answer 2

Two words: Sample Size...A power analysis is a must. By including a competent statistician on your team from the get-go, you will likely save yourself a great deal of frustration when you are writing the results and discussion sections of your manuscript or report.

It is all too common for a principal investigator to collect data prior to consulting with a statistician with the expectation of a "predictive model" or a "causal relationship" from a sample of less than 30 subjects. Had the PI consulted with a statistician prior to collecting data, the statistician would have been able to inform the PI, after appropriate analyses, to collect more data/subjects or to restructure the goals of their analysis plan/project.

Answer 3

I suppose it depends on how strictly you interpret the word "design". It is sometimes taken to mean completely randomized vs. randomized blocks, etc. I don't think I've seen a study that died from that. Also, as others have mentioned, I suspect "died" is too strong, but it depends on how you interpret the term. Certainly I've seen studies that were 'non-significant' (and that researchers subsequently did not try to publish as a result); under the assumption that these studies might have been 'significant' if conducted differently (according to obvious advice that I would have given), and hence been published, might qualify as "died". In light of this conception, the power issue raised by both @RobHall and @MattReichenbach is pretty straightforward, but there is more to power than sample size, and those could fall under a looser conception of "design". Here are a couple of examples:

• Not gathering / recording / or throwing away information
  I worked on a study where the researchers were interested in whether a particular trait was related to a cancer. They got mice from two lines (i.e., genetic lines, the mice were bred for certain properties) where one line was expected to have more of the trait than the other. However, the trait in question was not actually measured, even though it could have been. This situation is analogous to dichotomizing or binning a continuous variable, which reduces power. However, even if the results were 'significant', they would be less informative than if we knew the magnitude of the trait for each mouse.

  Another case within this same heading is not thinking about and gathering obvious covariates.

• Poor questionnaire design
  I recently worked on a study where a patient satisfaction survey was administered under two conditions. However, none of the items were reverse-scored. It appeared that most patients just went down the list and marked all 5s (strongly agree), possibly without even reading the items. There were some other issues, but this is pretty obvious. Oddly, the fellow in charge of conducting the study told me her attending had explicitly encouraged her not to vet the study with a statistician first, even though we are free and conveniently available for such consulting.

Answer 4

I've seen this kind of problem in survey-like and psychological experiments.

In one case, the entire experiment had to be chalked up to a learning experience. There were problems at multiple levels that resulted in a jumble of results, but results that seemed to give some support for the hypothesis. In the end, I was able to help plan a more rigorous experiment, which essentially had enough power to reject the hypothesis.

In the other case, I was handed a survey that had already been designed and executed, and there were multiple problems that resulted in several areas of interest being affected. In one key area, for example, they asked how many times the customers were turned away from an event due to it being full when they arrived. The problem is that there's no time range on the question so you couldn't tell the difference between someone who had tried to attend 4 times and been turned away 4 times and someone who had tried to attend 40 times and only been turned away 4 times.

I'm not a trained, capital-s Statistician, but if they'd come to me beforehand, I would have been able to help them fix these issues and get better results. In the first case, it still would have been a disappointing, "Sorry, your hypothesis seems extremely unlikely", but it could have saved them a second experiment. In the second case, it would have given them answers to some important questions and would have made the results sharper. (Another problem they had is that they surveyed multiple locations over time and at least some people were thus surveyed multiple times, with no question like "Have you taken this survey elsewhere?")

Perhaps not statistical issues per se, but in both of these cases smart, well-educated domain experts created instruments that were flawed, and the results were one dead experiment and one experiment with limbs amputated.