6
$\begingroup$

I am trying to understand the poem Hiawatha Designs an Experiment because it looks like the kind of in-joke that it would be nice to get, as a statistician. Here is what I would like to know:

Unhelpful answer: "The poem is about the bias-variance tradeoff."

Helpful answer: The poem is critical of the following kind of experimental design for the following reasons: ....

Great answer: Here is a detailed set of Cliff-like notes explaining all the terms used in the poem (Anderson, Bancroft, truncated normals, etc.) Here are some links to books or papers espousing the kind of experimental designs that are being criticised.

Ideal answer: Further to the info from "Great answer", here is an actual example of a study which might be criticised by the writer of the poem.

I'd also be really interested in some historical context. Is the poem commenting on some sort of trend in statistics which was later reversed? Or is it still relevant?

$\endgroup$

1 Answer 1

3
$\begingroup$

I will make a stab at a helpful answer (the teacher in me thinks that you will learn more by using google or wikipedia or other resources to learn about truncated normals and who Bancroft is than reading a Cliff's notes versions (and the realist in me realizes that I would have to look up Bancroft to be sure of giving a good answer, I probably will later)).

The poem is not critical of any specific experimental design, rather it is critical of focusing too much on one aspect of the research without considering other important questions.

I remember one friend (my professor at the time) telling me of the first experiment he designed after graduating, it was a beautiful experiment and would have given great answers to many questions if it had ever been carried out, but it was too expensive and too complicated so the people he designed it for never did it. At the time of telling the story many years later he talked about how he could see ways that he could have designed the experiment in ways that they would have easily run the experiment and while these other ways were inferior to his original design on paper, any experiment that yields actual data is better than one that is never run.

When I meet with a client at the beggining of a project the first hour us usually spent working out exactly what question(s) they are trying to answer. In some cases that is all I need to do, help them understand what their question really is, then the experiment and analysis is simple enough that they can do it themselves. While I really would like to do some of the complicated designs, more often I find myself helping them find a simpler design that answers a specific question well and is doable (I mostly do low budget (often no budget) projects). Hiawatha in the poem had his own goal of being unbiased, quick, and powerful, but did not take time to find out that the main goal was to actually hit the target. It is critical of using a design or technique because it is cool rather than finding out what the real question is and finding the design and technique that will answer that question.

I have often seen in students, new graduates, or clients that have had enough stats to be dangerous (I have been included in all these categories and was guilty of the same sins), a tendancy to do every analysis that the textbook/teacher taught. When I ask them why they did a particular test the responses are often along the lines of "that is what the text did" or " that is what they did in this paper", but when I ask what question that test answers I often get the deer in the headlights look (hopefully followed by enlightenment of why do a test if it does not answer a question you are interested in). I think Hiawatha falls in this category, he was doing fancy things for the sake of doing fancy things rather than focusing on what the goal really was (hit the target).

The poem is not saying that there are particular designs that are bad, for some questions the more complicated designs are appropriate, but for others the simpler designs are better. We need to find the right design for the question and the science involved, not focus on our favorite regardless of whether it is appropriate or not. Sometimes it is important to reduce or eliminate bias, other times a little bias can be lived with if it gains us something else such as reduced variance or reduction of costs to where things can be done (and my wife finds bias very useful, though to her bias means fabric that has been cut into diagonal strips which gives it more flexibility and stretch).

Hopefully this is more helpful than just rambling. Maybe this will remind others of specific cases that they can add where a particular tool that could be useful in other cases was missused in a particular case.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.