Clinical trials significance I have been a studying high energy physics (HEP) for the last few years but I recently started working on a project in medical imaging. I have been a little surprised (not entirely I was aware that 95% was commonly used) to find major studies reporting using 95% cl. In HEP the convention is not worry too much until 3 sd if not 5. I appreciate in the 'real world' such idealistic situations cannot always be created but even so 95% doesn't seem such a high level of confidence. What is the rationale behind this? Is it simply a pragmatic one in the interests of completing in a timely manner? I have already come across trial or two which appear, to say the least, couterintuitive in their findings.
Can anyone recommend a good book to jump into the mathematics of this? I have borrowed a fairly qualitative introduction but things like power calculations and kaplan meier curves are essentially totally new to me.
 A: There are a few reasons:
1) If you decrease the chance of type I error you increase the chance of type II error. Sometimes one is more important, sometimes the other. Often, I think, 5% is too low because it increases type II error.
2) In some ways, type I errors never happen. That is, suppose your null hypothesis is that two means are equal. But, in a population, two means are never exactly equal. It's more a question of how far apart they are.
3) It's traditional and there's no really great reason to change it. 
4) It's traditional and journal editors/dissertation committees/pointy haired bosses demand it.
5) It's traditional and it lets you avoid thinking about whether it makes sense in a particular situation. 
A: it should be pointed out that in drug development the evidence accrues across phase i - phase iii clinical trials. Although, if we're not bayesian then the phase iii is a stand alone trial. In fact the fda wants two phase iii studies showing statistical significance: http://onlinelibrary.wiley.com/doi/10.1002/(SICI)1097-0258(19980815/30)17:15/16%3C1813::AID-SIM983%3E3.0.CO;2-8/full. Stephen Senn points out that this implies a p-value of 1/1600 (see Senn's book statistical issues in drug development, sectn 12.2.8 the two-trials rule). ipso facto an inefficacious drug will not reach market
A: Yes, the way in which medical and social researchers use statistics has a number of problems. Significance testing is probably inappropriate in the vast majority of cases and in my opinion has possibly led to stagnation of these fields. 
I have come to think that instead of just seeing if two averages are different, the researchers should be focused on describing the distribution of the results in search of subgroups, estimating parameters, and guessing mathematical models that may explain the data generating process at the individual level.
My personal guess would be that well over 50% of the conclusions drawn in the literature are false or at least only true for an extremely narrow set of conditions that will never be reproduced exactly again. Highly recommended reading:
Meehl, P.E. (1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34:103-115.
There is empirical evidence of serious problems as well:
Recently researchers from Amgen have claimed they could replicate 5/52 results from "landmark studies".
Researchers from Bayer reported slightly better success in attempting to validate new drug targets, replicating 19/67 results.
An effort by John Ioannidis to reproduce data on the presence of sex difference for various diseases reported replicating 1/432 results.
Edit:
To clarify a bit. I do not think significance testing the difference between groups is bad in and of itself, it can clearly provide information on what may be worth investigating further. It is just that it provides so little information in comparison with the alternatives so should always be done in conjunction. The real world result has been that all focus goes on the significance test. This is at the expense of the other approaches to analyzing data mentioned, and it is lack of the others that I believe has stagnated medical research.
