Following ICH-E9 guideline on statistical principles for clinical trials,

A statistical analysis plan is a document that contains a more technical and detailed elaboration of the principal features of the analysis described in the protocol, and includes detailed procedures for executing the statistical analysis of the primary and secondary variables and other data.

A SAP is also used beyond clinical trials in other fields of biostatistics. There are no strict regulatory authorities insisting on proper SAPs, yet the appropriate procedures can be even less obvious than in clinical trials, thus clarification in a SAP is even more necessary.

How deep should one go into detail there? I'm taught a certain way to think by first understanding how the (mostly qualitative) question of the scientist is expressed by (quantitative) mathematical formulas (in German: "Operationalisierung der Fragestellung"), then to identify with him anything possibly confounding, then to fill in more and more assumptions (beginning with the most plausible ones) until the question is statistically tractable (and weaknesses of the chosen procedure are best visible).

Are there publicly available resources and good examples of statistical analysis plans (SAP) for various fields of science? What am I allowed to omit in a SAP although I thought about it for designing the analysis?

  • 1
    $\begingroup$ Might help to make this of more general interest by explaining what the SAP is for - who reads it? what decisions are based on it? what are the consequences of deviating from it if the trial is performed?, &c. $\endgroup$ Oct 10 '14 at 22:54
  • $\begingroup$ Feel free to improve my question! $\endgroup$ Oct 12 '14 at 20:56
  • $\begingroup$ I've no idea myself what it's for, never having read ICH-E9. $\endgroup$ Oct 13 '14 at 8:59
  • $\begingroup$ I don't know it is not offtopic but you could check a book by George Casella on statistical design: springer.com/statistics/statistical+theory+and+methods/book/… $\endgroup$
    – Tim
    Dec 3 '14 at 8:04

While I would argue that a statistical analysis plan (SAP) is a really good idea for any study or experiment, it is in clinical research that you will find most guidance. That is because the field is heavily regulated and because it is in the interest of industry to describe best practices.

In most such research, there is already a history and a huge set of accepted methodology. That is why you can find documents that lay out the structure of the SAP in detail, such as:

There are many excellent resources online for evaluating or constructing the SAP. These include many articles and sites offering criteria for assessing or evaluating SAPs --- for example, this Review of Statistical Analysis Plans.

Most companies and larger research institutions have templates for various documents, including the SAP --- this sample Word template is one of many.

Also, in such settings, you could expect the SAP to undergo just as rigorous a review as the protocol, case report forms, data management plan, statistical programs, and every other artifact associated with the research.

At its heart, the SAP is actually quite a practical document. It allows communication among many different players about what may be expected from the research. It is not going to be the love-child of academia and industry, though.

Most of the SAP is probably going to be devoted to very mundane things such as planned data listings, summary tables, and summary figures. The actual statistical analyses planned might occupy only a little bit of text in proportion to the rest of the document.

However, really good listings, summary tables, and summary figures can be very informative.

How deep should one go into detail there?

Depending on the type of study, you may wish to go into more or less detail. However, there is a line to walk. If you over-specify the statistical analysis, you may commit to things that cannot be done, depending on the actual distribution of the data that is collected. Also, it does no good to specify highly sophisticated statistical analysis that may be quite "brittle" to real-world exigencies, or that might pose a problem in communication with regulatory authorities.

There will probably need to be some sound basis given for the choice of statistical methods used. This will most likely be in the form of some more or less standard references depending a bit on the type of research.

While I really agree with your general approach of starting with first principles, in practice it is usually better to use an analysis that is as standard and as simple as can meet the needs of the study. Then, if resources and the data permit, more elaborate analyses could be planned.

This means that usually there needs to be statistical input into the design of the study, in order to allow a decently straightforward statistical analysis in the first place.

What am I allowed to omit in a SAP although I thought about it for designing the analysis?

That is a good question. I would lean toward not putting in a lot of "sophisticated" analysis methods or branching logic, unless for some reason that was deemed absolutely necessary. Instead, you might wish to keep track of the ideas you have for extra analysis for the case where you have adequate time at hand.

If you can foresee bad things happening with respect to the distribution of the data, then perhaps the research should be redesigned or not be carried out. If you can foresee certain common problems, then you can certainly acknowledge them along with including a bit of discussion about how they will be handled.

For example, it is pretty common to provide that a nonparametric test will be used instead of a parametric test, if the data distribution requires.

In the main, you want to provide a straightforward description of the proposed statistical methods, with perhaps some discussion of why they are appropriate. But, if a lot of text is needed, I would rethink the analysis or the research.

If you over-promise results, then you may not be able to deliver on them. If you under-specify the analysis, then you may create unnecessary negative feedback or delays.

As mentioned, there are many online resources for SAPs used in clinical research. Most of those principles and practices are just as useful in every other field of research, from my experience. However, it is unrealistic to expect people to put the same amount of effort into the SAP unless perhaps the research will cost a lot of money and time. Even then...


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.