2
$\begingroup$

I have been a studying high energy physics (HEP) for the last few years but I recently started working on a project in medical imaging. I have been a little surprised (not entirely I was aware that 95% was commonly used) to find major studies reporting using 95% cl. In HEP the convention is not worry too much until 3 sd if not 5. I appreciate in the 'real world' such idealistic situations cannot always be created but even so 95% doesn't seem such a high level of confidence. What is the rationale behind this? Is it simply a pragmatic one in the interests of completing in a timely manner? I have already come across trial or two which appear, to say the least, couterintuitive in their findings.

Can anyone recommend a good book to jump into the mathematics of this? I have borrowed a fairly qualitative introduction but things like power calculations and kaplan meier curves are essentially totally new to me.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ The reason for $1-\alpha$ = 95% CI's is largely the same as for $\alpha$ = 5% significance levels. $\endgroup$
    – Glen_b
    Dec 16, 2013 at 23:40
  • $\begingroup$ Very interesting. I'm a bit concerned by the idea that it is used because it is achievable, I can be pragmatic up to a point, but surely an experiment should aim for a minimum level of evidence not just do what is necessary to reach 95% (p=0.05) within reason. I wouldn't go as far as extraordinary claims require extraordinary evidence but if one is going to claim to disprove a long standing theory, discover a new particle/cure a greater level of significance that a fit for some calibration curve seems like a good idea. $\endgroup$
    – user36288
    Dec 17, 2013 at 0:03
  • 1
    $\begingroup$ I'm not sure what you mean by 'it's used because it's achievable'. That's not the message I take away from the linked page. I don't think anybody is saying that 5% should be used for any particular case (are they?), and certainly not for the case where one is trying to "disprove a long standing theory". You seem to be arguing against something nobody (to my knowledge) is saying. The convention in HEP is just as much a convention as the convention you see in say agricultural research, the distinction is that the stakes are different so of course suitable α (& β) is too. What did I miss? $\endgroup$
    – Glen_b
    Dec 17, 2013 at 1:18
  • 1
    $\begingroup$ If anything, there's a good argument that hypothesis testing should be used much more rarely than it is, and in many ways its use in HEP is bizarre... even more bizarre than in many other areas. Why aren't they using Bayesian statistics, one might reasonably ask. I am not sure. It seems at least as counterintuitive to use $5 \sigma$ in HEP as to use say $2 \sigma$ when choosing between old and new varieties of tomato. $\endgroup$
    – Glen_b
    Dec 17, 2013 at 1:29
  • $\begingroup$ @Glen_b I have wondered that myself about the Higgs Boson experiments. This was a case in which they expected to find a "peak" signal within a certain range, shouldn't the null hypothesis have been the prediction? I suppose maybe the theory did not predict well enough for a point null. $\endgroup$
    – Flask
    Dec 17, 2013 at 4:47

3 Answers 3

Reset to default
4
$\begingroup$

There are a few reasons:

1) If you decrease the chance of type I error you increase the chance of type II error. Sometimes one is more important, sometimes the other. Often, I think, 5% is too low because it increases type II error.

2) In some ways, type I errors never happen. That is, suppose your null hypothesis is that two means are equal. But, in a population, two means are never exactly equal. It's more a question of how far apart they are.

3) It's traditional and there's no really great reason to change it.

4) It's traditional and journal editors/dissertation committees/pointy haired bosses demand it.

5) It's traditional and it lets you avoid thinking about whether it makes sense in a particular situation.

Improve this answer
$\endgroup$
2
  • $\begingroup$ 1) is a good point. i guess not something i've encountered much. $\endgroup$
    – user36288
    Dec 17, 2013 at 0:35
  • $\begingroup$ In medical research getting adequate numbers of controlled observations is always a big challenge, emphasizing the importance of point (1). $\endgroup$
    – EdM
    Dec 17, 2013 at 17:13
1
$\begingroup$

it should be pointed out that in drug development the evidence accrues across phase i - phase iii clinical trials. Although, if we're not bayesian then the phase iii is a stand alone trial. In fact the fda wants two phase iii studies showing statistical significance: http://onlinelibrary.wiley.com/doi/10.1002/(SICI)1097-0258(19980815/30)17:15/16%3C1813::AID-SIM983%3E3.0.CO;2-8/full. Stephen Senn points out that this implies a p-value of 1/1600 (see Senn's book statistical issues in drug development, sectn 12.2.8 the two-trials rule). ipso facto an inefficacious drug will not reach market

Improve this answer
$\endgroup$
1
  • $\begingroup$ Or at least it's usually two separate Phase 3 trials that need to achieve $p\leq0.05$ (two-sided), so stronger (in the sense that two arbitrary splits of the data need to agree, too) one-sided $p\leq 0.025^2$. $\endgroup$
    – Björn
    Dec 25, 2017 at 14:22
0
$\begingroup$

Yes, the way in which medical and social researchers use statistics has a number of problems. Significance testing is probably inappropriate in the vast majority of cases and in my opinion has possibly led to stagnation of these fields.

I have come to think that instead of just seeing if two averages are different, the researchers should be focused on describing the distribution of the results in search of subgroups, estimating parameters, and guessing mathematical models that may explain the data generating process at the individual level.

My personal guess would be that well over 50% of the conclusions drawn in the literature are false or at least only true for an extremely narrow set of conditions that will never be reproduced exactly again. Highly recommended reading:

Meehl, P.E. (1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34:103-115.

There is empirical evidence of serious problems as well:

Recently researchers from Amgen have claimed they could replicate 5/52 results from "landmark studies". Researchers from Bayer reported slightly better success in attempting to validate new drug targets, replicating 19/67 results. An effort by John Ioannidis to reproduce data on the presence of sex difference for various diseases reported replicating 1/432 results.

Edit: To clarify a bit. I do not think significance testing the difference between groups is bad in and of itself, it can clearly provide information on what may be worth investigating further. It is just that it provides so little information in comparison with the alternatives so should always be done in conjunction. The real world result has been that all focus goes on the significance test. This is at the expense of the other approaches to analyzing data mentioned, and it is lack of the others that I believe has stagnated medical research.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.