My team and me would like to give a presentation to the non-statisticians of the company about the utility of the design of experiments. These non-statisticians are also our clients and they usually don't consult us before collecting their data. Do you know some real examples which would well illustrate Fisher's famous quote "To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he only may be able to say what the experiment died of." ? Preferably we are looking for an illustration in an industrial/pharmaceutical/biological context. We think of an example of an inconclusive statistical analysis which could have been successful if it had been preliminary well designed, but maybe there are other possible illustrations.

  • 6
    $\begingroup$ There are some on p47 onwards of Common Errors in Statistics (and How to Avoid Them) by Phillip Good, & James Hardin $\endgroup$ – onestop Jul 10 '12 at 15:04
  • $\begingroup$ Thanks. The previous boss of my team probably has this book. $\endgroup$ – Stéphane Laurent Jul 10 '12 at 17:36
  • $\begingroup$ @onestop I have the book in my hands. What is the chapter your are talking about ? I have the second edition of the book and there's nothing on p47. $\endgroup$ – Stéphane Laurent Jul 11 '12 at 9:02
  • $\begingroup$ Hmm, seems I was looking at the 4th edition on Google Books link above. There's a section entitled 'Experimental Design' in chapter 3, 'Collecting Data'. $\endgroup$ – onestop Jul 11 '12 at 12:28

I've run into designs where the experimenter wanted to test between subject effects but the design was more suitable for within subject effects.

For example, one experiment consisted of 8 rats, four on diet A and four on diet B, and the weight of the rat was measured each day for four weeks. This was fine if they were interested in the time effect of each diet but the goal was to investigate differences in the diets.

They thought by measuring each rat 28 times they had lots of data, but the experimental unit for the diet effect was the rat, which they only had 4 for each treatment. They could have measured the rats 10 times a days but it would have made no difference, in the end they needed more rats.

  • 5
    $\begingroup$ (+1) I suspect medical research stands nearly alone in human endeavors with regard to the need and desire to have more rats. $\endgroup$ – cardinal Jul 10 '12 at 17:59
  • 1
    $\begingroup$ I get involved in a lot of laboratory experiments at Lankenau and the sample sizes are always small and involve mice or rats. $\endgroup$ – Michael Chernick Jul 10 '12 at 18:15
  • $\begingroup$ When doing lab experiments the animals are often sacrificed and I think that is one reason why they try to keep the number of animals as small as possible. But in such circumstances you would want to take enough to rach meaningful conclusions. $\endgroup$ – Michael Chernick Jul 10 '12 at 18:34

I did some work for a organization called the National Foundation for Celiac Awareness. The organization promotes the public's awareness of Celiac Disease and provides a checklist of symptoms of the disease which involves intolerance to foods containing gluten. They conducted a survey on the internet by just opening it up to anyone who wanted to participate. Over the years they collected thousands of responses from the public. However they were hoping to draw conclusions about the general public based on the survey results. I had to tell them that the respondents were selfselected rather than random and this could create bias. Since the degree of bias is unknown we could not do any inference inspite of the large amount of data.

Now the respondents seemed to be a peculiar group. Many are very serious and answered to express concern that they or a relative might have the disease. But there also were a distnct number of people answering in a wise-guy fashion. This was obvious from the fake names, strange email addresses and postal addresses that they provided with their answers.

I felt that the data was only useful in an exploratory sense and the frequency of responses might be useful for fomrulating hypotheses that could be tested in a well planned future survey. But thus far my advice has not been heeded and they are running another one of these easy to do self selecting surveys on the internet.

  • $\begingroup$ (+1) Good example. Sometimes clients collect very specific samples but they want to make conclusions about the whole population. $\endgroup$ – user10525 Jul 10 '12 at 15:05
  • $\begingroup$ Thanks for this interesting example (but it is not appropriate for my non-statisticians colleagues) $\endgroup$ – Stéphane Laurent Jul 10 '12 at 17:34
  • $\begingroup$ @StéphaneLaurent Yes isn't it? it has to do with poor design for a medical study. $\endgroup$ – Michael Chernick Jul 10 '12 at 18:01
  • $\begingroup$ Yes Michael but my clients never conduct a survey. $\endgroup$ – Stéphane Laurent Jul 10 '12 at 18:02
  • $\begingroup$ @StéphaneLaurent The idea is the principle of bias due to lack of randomization. It applies to experiments and surveys in very much the same way. $\endgroup$ – Michael Chernick Jul 10 '12 at 18:12

Some time ago I was asked to analyze the results of an experiment on how the night storage position of a photovoltaic solar array affected the rate at which soil accumulated on the array. (These large concentrating photovoltaic arrays track the sun all day, but at night they are typically stored pointing straight up, as this is the minimum stress position for the tracker.) Soiling is a big issue, because it significantly reduces energy production, and cleaning is not cheap. The experiment had been run on a field of roughly 120 trackers; the west half had been stowed vertically and the east half horizontally (this aligned with the tracker connections to the two inverters, which would convey an advantage in energy production during the experiment if there is a significant effect and no particular pattern of soiling otherwise, so it's not, by itself, a bad idea.)

Unfortunately, there is a strong prevailing wind pattern across the desert from the south-southwest, and a large building to the south of the western part of the field, "shading" (somewhat) much of the western part of the field from windblown particulates. Additionally, trackers "shade" each other from the wind to some extent. Consequently, the mechanisms by which soil accumulates (e.g., wind-blown or settling) vary in relative magnitude across the field. This in turn implies that arrays accumulate soil at different rates dependent upon location; this is not a small effect.

The end result of the analysis was, essentially, that it wasn't implausible that storage position made a difference, but we could not, by any means, rule out the possibility that the effect was trivial, nor determine with any great confidence (based on the data) the sign of the effect. I then designed a followup experiment, assigning storage positions based on array location with the objective of being able to estimate the soiling "response surface" across the field for both storage positions, estimating "settling" vs "wind-blown" soiling rates, and of course the effect of storage angle on both of these. This experiment was quite successful and we were able to obtain a clear picture of the benefits of vertical stow after just a couple of months.


I was asked by a colleague to 'do the stats' on a study looking at the correlation between a certain type of weather event and failures in a type of infrastructure that are typically attributed to simple wear and tear. The colleague wanted to see if the weather events were actually contributing to the failure or not. A team of people had already spent a lot of time and effort collecting a vast amount of data and the research paper was pretty much finished, they just needed someone to 'do the stats' and fill in the final bit of the results section.

The problem was, they had painstakingly ensured that the data set contained only 'interesting' periods in which the weather event in question had occurred. That meant there was no way to compare the failure rate during events with non-event times. I tried repeatedly to explain the problem, but they were never really convinced, because the simply had so much data that surely I could get something out of it.

Fortunately there was still a range of severity of the weather events and there was a weak correspondence between the severity and failure rate, so we salvaged something from it at least, but the result could have been so much more definitive had they thought about how to 'do the stats' before embarking on the data collection exercise.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.