References for how to plan a study

Question

In an average (median?) conversation about statistics you will often find yourself discussing this or that method of analyzing this or that type of data. In my experience, careful study design with special thought with regards to the statistical analysis is often neglected (working in biology/ecology, this seems to be a prevailing occurrence). Statisticians often find themselves in a gridlock with insufficient (or outright wrong) collected data. To paraphrase Ronald Fisher, they are forced to do a post-mortem on the data, which often leads to weaker conclusions, if at all.

I would like to know which references you use to construct a successful study design, preferably for a wide range of methods (e.g. t-test, GLM, GAM, ordination techniques...) that helps you avoid pitfalls mentioned above.

Answer 1

1. I agree with the point that statistics consultants are often brought in later on a project when it's too late to remedy design flaws. It's also true that many statistics books give scant attention to study design issues.

2. You say you want designs "preferably for a wide range of methods (e.g. t-test, GLM, GAM, ordination techniques...". I see designs as relatively independent of statistical method: e.g., experiments (between subjects and within subjects factors) versus observational studies; longitudinal versus cross-sectional; etc. There are also a lot of issues related to measurement, domain specific theoretical knowledge, and domain specific study design principles that need to be understood in order to design a good study.

3. In terms of books, I'd be inclined to look at domain specific books. In psychology (where I'm from) this means books on psychometrics for measurement, a book on research methods, and a book on statistics, as well as a range of even more domain specific research method books. You might want to check out Research Methods Knowledge Base for a free online resource for the social sciences.

4. Published journal articles are also a good guide to what is best practice in a particular domain.

Answer 2

My rule of thumb is "repeat more than you think it's sufficient".

Answer 3

In general, I would say any book that has DOE (design of experiments) in the title would fit the bill (and there are MANY).

My rule of thumb for such resource would be to start with the wiki page, in particular to your question, notice the Principles of experimental design, following Ronald A. Fisher

But a more serious answer would be domain specific (clinical trial has a huge manual, but for a study on mice, you'd probably go with some other field-related book)

Answer 4

Answering with an aphorism, I believe that your study design will be successful as soon as it actually exists in its full-fledged form. The game of reviewing as it is played in academia is primarily a game of academics showing to each other that they have not completed that step in its full depth, e.g. by violating assumptions or omitting biases where they should be expected. If study design is a skill, it's the skill of making your research bulletproof to these critics.

Your question is very interesting but I am afraid that there is no short answer. To the best of my knowledge, the only way to learn thoroughly about research designs, whether experimental or observational, is to read the literature in your field of specialisation, and then to go the extra mile by connecting with academics in order to learn even more on how they work, in order to, eventually, write up your own research design.

In my field (European political science), we generically offer "research design" courses that span over all types of studies, but even then we miss important trends and also lack a deep understanding of our methods. After taking at least three of these courses, I have become convinced that no academic resource can replace learning from other academics, before confronting real-world settings directly.

I guess that your field also has these 'methods journals' that can be as painfully boring and complex to the outsider than they are helpful and interesting to actual 'study designers' -- and so would recommend that you start digging this literature first, eventually tracking down the recurring bibliographic items that might help you most with study design in biology/ecology. Google Scholar definitely flags a few books with the words 'ecology research methods'.

Answer 5

This might not be 100% what you are looking for, but I can name a few books that span both, quantitative and qualitative research designs in social sciences. (Personally, I find it very helpful to have the full options at hand to adapt the design to your research question, the existing knowledge in the field, the unit of comparison, the accessibility of data.)

• Gschwend, T., & Schimmelfennig, F. (Eds.). (2007). Research Design in Political Science: How to Practice What They Preach. Houndsmill, et al.: Macmillan.

• Leavy, P. (2017). Research Design: Quantitative, Qualitative, Mixed Methods, Arts-based, and Community-based Participatory Research Approaches. [S.l.]: Guildford.

(Further off topic to the original questions, but potentially helpful for readers of this questions who want to learn more about the underlying logic of research designs: The grande debate about the commonalities and differences in research designs in qualitative and quantitative modes of enquiry. The first book basically sketches out how qualitative research could, should and is following causal logic, the second provides a bit "Yes, but..." and has concluding and synthesizing chapters.)

• King, G., & Keohane, R. O. V. S. (1994). Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton, New Jersey.

• Brady, H. E., & Collier, D. (Eds.). (2004 // 2010). Rethinking Social Inquiry: Diverse Tools, Shared Standards (2. ed.). Lanham et al.: Rowman & Littlefield Publishers.