I've observed that many experimental labs in my field don't do a power analysis before they start collecting data. In practice, it seems that many researchers collect data and periodically run statistical tests (e.g. a t-test) to see if they gained significance. Once they obtain a significant p-value, they stop collecting data.
I'm sure most visitors to this site will immediately realize the problem with this, and to be fair many good experimentalists in my field are aware of this as well. The problem is that you are likely to observe a significant result just by pure chance if you run enough tests. Some people will collect "an extra few data points" and re-run the test to be sure it is significant, but is this a viable solution?
I have a few general questions related to this problem:
- Are there good papers/references for me to read on this problem? Are there some quantitative principles that can help us interpret these situations?
- What advice should I give to a student working on a project like this? What should they be doing to avoid this problem? Is doing a power-analysis really enough? What happens when they do a rough power analysis, do the experiments and find p = 0.06?
- The two above questions basically ask for the set of "best practices" for a researcher to follow. If there is a set of agreed upon best practices, how can I convince/persuade people to follow them? What intuitive logic can be used? Any nice visualizations/diagrams?
Feel free to comment on any of the above. I presume that there is a lot of work on this problem (esp. in the context of clinical trials?), so I apologize if this question is a bit naive. You can just direct me to papers if that is the case.
Update: A brief literature search provided the following references:
I'm still interested in how to digest and distill this work into something that biologists can use and understand.