27
$\begingroup$

I've come across this quote numerous times:

To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of. -- Ronald Fisher (1938)

To me, it seems perhaps a little presumptuous. The only examples I've ever found describing how experiments die without good design are around lack of controls, or poor controls. For example, experiments that control for the application of a fertilizer, but fail to control for the environment required for the application. Maybe it's just me, but it seems that a quick read through the Wikipedia section on Fisher's design principles would cover most bases.

As a statistician, how often do you see design of experiment-related problems with data? Are they always related to those few factors mentioned by Fisher, or or there other serious pitfalls that we non-statistically trained scientists should be looking out for?

$\endgroup$
  • 4
    $\begingroup$ How often: very often. To call the experiment "dead" is usually going too far, but I many experiments I see could have been much better with only slight changes in the design. $\endgroup$ – markseeto Aug 21 '13 at 9:52
  • 3
    $\begingroup$ I've seen a few. While it might be presumptuous now, remember that when Fisher said it, you couldn't just look up wikipedia. The rate may have been far higher in the early days. $\endgroup$ – Glen_b Aug 21 '13 at 10:22
  • 4
    $\begingroup$ Nice that you raise this point. I'm also curious about what perhaps may be the first time I've seen a quadruple qualifier: "To me, it seems perhaps a little presumptuous." :-) $\endgroup$ – rolando2 Aug 21 '13 at 17:03
  • 1
    $\begingroup$ @rolando2: Heh, well it is Fisher. He earned all those qualifiers :D $\endgroup$ – naught101 Aug 21 '13 at 22:16
  • 5
    $\begingroup$ I have seen--literally--many thousands of datasets in my career (and virtually none of them were collected according to a design reviewed by any statistician). Most of those were collected for formal purposes, such as satisfying regulatory requirements. I cannot recall a single one that did not have some design-related problems (although sometimes these were minor). This is not to say the datasets were useless or "dead": but in almost all cases my task was (to continue the medical analogy) first to resuscitate the dataset and then to apply it to its intended purpose, if at all possible. $\endgroup$ – whuber Aug 23 '13 at 14:41
14
$\begingroup$

I believe what Fisher meant in his famous quote goes beyond saying "We will do a full factorial design for our study" or another design approach. Consulting a statistician when planning the experiment means thinking about every aspect of the problem in an intelligent way, including the research objective, what variables are relevant, how to collect them, data management, pitfalls, intermediate assessment of how the experiment is going and much more. Often, I find it is important to see every aspect of the proposed experiment hand-on to really understand where the difficulties lie.

My experience is mainly from medical applications. Some of the issues I have encountered that could have been prevented by consulting a statistician beforehand:

  • Insufficient sample size is, of course, number one on this list. Often, data from previous studies would have been available and it would have been easy to give a reasonable estimate of the sample size needed. In these cases, the only recourse is often to do a purely descriptive analysis of the data and promise further research in the paper (not publishing is usually not an option after doctors invested valuable time).
  • Execution of the experiments is left to convenience and chance instead of design. An example I am currently working on has measurements collected over time. The measurement times, measurement frequency and end of monitoring period all vary wildly between individuals. Increasing the number of measurements per individual and fixing the measurement dates and end of monitoring period would have been fairly little extra work (in this case) and would have been very beneficial to the study.
  • Poor control of nuisance factors that could have easily been controlled. E.g. measurements were sometimes performed on the day of sample collection and sometimes later, leaving the possibility that the sample has degraded.
  • Poor data management, including my personal favourite "I rounded the data before putting it into the computer, because the machine is inaccurate in its measurements". Often, relevant data is just not collected and it is impossible to get it after the fact.

Often, problems with a study go even further back, to the initial conception of the research:

  • Data is sometimes collected without a clear objective and just the assumption that it will be useful somehow. Producing hypotheses and "significant results" is left to the statistician.
  • And the opposite: data is scraped together with the aim of proving a specific point that the PI has in his head, irrespective of the data and what can actually be proved with it. This time, the statistician is just supposed to put his stamp of significance on pre-written conclusions without the conclusions getting adjusted in the face of the data.

So far, this mainly sounds like the statistician suffers and maybe scientific integrity suffers when the PI tries to push conclusions not supported by the data (always a fun discussion). But the experimental team suffers as well, because they do unnecessary extra work (while not doing necessary work) during the experimental phase and need to spend much more time in discussion with their statistician after the fact, because they did not get their advice before. And of course, the final paper will be worse, will have fewer conclusions (and more "conjectures") and will likely not make it into that high-impact journal the PI wanted.

$\endgroup$
  • $\begingroup$ With regard to the second of your 2nd set of bullet points, I think the normal rationale of a study is to gather data with aim of proving specific points. $\endgroup$ – Robert Jones Aug 21 '13 at 11:16
  • 1
    $\begingroup$ You are, of course, completely right. I was a bit too short there. What I meant to mention was a scenario where a PI who is very determined to prove a point and poor quality data that cannot prove that point (often due to fundamental design issues) get together. $\endgroup$ – Rob Hall Aug 21 '13 at 11:20
12
$\begingroup$

Two words: Sample Size...A power analysis is a must. By including a competent statistician on your team from the get-go, you will likely save yourself a great deal of frustration when you are writing the results and discussion sections of your manuscript or report.

It is all too common for a principal investigator to collect data prior to consulting with a statistician with the expectation of a "predictive model" or a "causal relationship" from a sample of less than 30 subjects. Had the PI consulted with a statistician prior to collecting data, the statistician would have been able to inform the PI, after appropriate analyses, to collect more data/subjects or to restructure the goals of their analysis plan/project.

$\endgroup$
  • 1
    $\begingroup$ I disagree with "A power analysis is a must". I think a lot of people overstate the importance of power analysis. $\endgroup$ – markseeto Aug 21 '13 at 9:48
  • 3
    $\begingroup$ @mark999: Could be, but it doesn't negate the importance of performing some sort of power analysis before doing the experiment, which I understand to be Matt's point. $\endgroup$ – Scortchi Aug 21 '13 at 11:55
  • 3
    $\begingroup$ @mark999: They can turn out to be useful, of course. But under what circumstances would you not recommend performing any kind of power analysis (I'm including estimating the expected width of confidence intervals) before doing an experiment? I can only think of (1) a pilot study, where you're only interested in running through the protocol & roughly estimating the error, & (2) an experiment for which you can't choose a sample size for some reason, making power analysis redundant. $\endgroup$ – Scortchi Aug 21 '13 at 22:58
  • 2
    $\begingroup$ @mark999: I think we do. For your case (B), I'd suggest pilot study -> power analysis -> experiment to test hypotheses or estimate effect sizes as an unimpeachable plan. $\endgroup$ – Scortchi Aug 22 '13 at 8:23
  • 3
    $\begingroup$ Even if you have a fixed sample size I don't see any reason to bury your head in the sand and avoid a power analysis (reasonable responses to resource constraints and ignorance aside). $\endgroup$ – Andy W Aug 23 '13 at 13:27
11
$\begingroup$

I suppose it depends on how strictly you interpret the word "design". It is sometimes taken to mean completely randomized vs. randomized blocks, etc. I don't think I've seen a study that died from that. Also, as others have mentioned, I suspect "died" is too strong, but it depends on how you interpret the term. Certainly I've seen studies that were 'non-significant' (and that researchers subsequently did not try to publish as a result); under the assumption that these studies might have been 'significant' if conducted differently (according to obvious advice that I would have given), and hence been published, might qualify as "died". In light of this conception, the power issue raised by both @RobHall and @MattReichenbach is pretty straightforward, but there is more to power than sample size, and those could fall under a looser conception of "design". Here are a couple of examples:

  • Not gathering / recording / or throwing away information
    I worked on a study where the researchers were interested in whether a particular trait was related to a cancer. They got mice from two lines (i.e., genetic lines, the mice were bred for certain properties) where one line was expected to have more of the trait than the other. However, the trait in question was not actually measured, even though it could have been. This situation is analogous to dichotomizing or binning a continuous variable, which reduces power. However, even if the results were 'significant', they would be less informative than if we knew the magnitude of the trait for each mouse.

    Another case within this same heading is not thinking about and gathering obvious covariates.

  • Poor questionnaire design
    I recently worked on a study where a patient satisfaction survey was administered under two conditions. However, none of the items were reverse-scored. It appeared that most patients just went down the list and marked all 5s (strongly agree), possibly without even reading the items. There were some other issues, but this is pretty obvious. Oddly, the fellow in charge of conducting the study told me her attending had explicitly encouraged her not to vet the study with a statistician first, even though we are free and conveniently available for such consulting.

$\endgroup$
  • $\begingroup$ Whoa... with the first one, what did they measure? that seems a little, um, obvious. Or were they given assurances before-hand that the traits were different in the different lines? Second example is cool, a kind of randomisation that most people wouldn't think about. $\endgroup$ – naught101 Aug 24 '13 at 5:52
  • 5
    $\begingroup$ It was just testing 1 strain vs the other. The trait in question really does tend to be higher for one of the lines, but there is some overlap--the distributions aren't totally separated. $\endgroup$ – gung Aug 24 '13 at 12:54
  • $\begingroup$ I had a similar experience to point 1: a microfluidic device was set up to a recognize certain type of cell. A mixture of cells-to-be-recognized and control cells was injected and a video stream + signal stream to be used for the recognition were acquired. Unfortunately, while the video stream could be used as reference for whether there was a cell at the detector at a given moment, there was no way to tell what type the cell actually was, so no way to determine whether a signal was true positive or false negative or no signal was true negative or false positive... $\endgroup$ – cbeleites Jul 12 at 14:11
8
$\begingroup$

I've seen this kind of problem in survey-like and psychological experiments.

In one case, the entire experiment had to be chalked up to a learning experience. There were problems at multiple levels that resulted in a jumble of results, but results that seemed to give some support for the hypothesis. In the end, I was able to help plan a more rigorous experiment, which essentially had enough power to reject the hypothesis.

In the other case, I was handed a survey that had already been designed and executed, and there were multiple problems that resulted in several areas of interest being affected. In one key area, for example, they asked how many times the customers were turned away from an event due to it being full when they arrived. The problem is that there's no time range on the question so you couldn't tell the difference between someone who had tried to attend 4 times and been turned away 4 times and someone who had tried to attend 40 times and only been turned away 4 times.

I'm not a trained, capital-s Statistician, but if they'd come to me beforehand, I would have been able to help them fix these issues and get better results. In the first case, it still would have been a disappointing, "Sorry, your hypothesis seems extremely unlikely", but it could have saved them a second experiment. In the second case, it would have given them answers to some important questions and would have made the results sharper. (Another problem they had is that they surveyed multiple locations over time and at least some people were thus surveyed multiple times, with no question like "Have you taken this survey elsewhere?")

Perhaps not statistical issues per se, but in both of these cases smart, well-educated domain experts created instruments that were flawed, and the results were one dead experiment and one experiment with limbs amputated.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.