15
$\begingroup$

I often use a 90% confidence level, accepting that this has a greater degree of uncertainty than 95% or 99%.

But are there any guidelines on how to choose the right confidence level? Or guidelines for the confidence levels used in different fields?

Also, in interpreting and presenting confidence levels, are there any guides to turn the number into language? For example, such as guides like this for Pearson's r (edit: these descriptions are for social sciences):

http://faculty.quinnipiac.edu/libarts/polsci/Statistics.html (page unresponsive on 26.12.2020)

Update

Thanks for the answers below. They were all VERY helpful, insightful and instructive.

In addition, below are some nice articles on choosing significance level (essentially the same question) that I came across while looking into this question. They validate what is said in the answers below.

Improve this question
$\endgroup$

3 Answers 3

Reset to default
11
$\begingroup$

In addition to Tim's great answer, there are even within a field different reasons for particular confidence intervals. In a clinical trial for hairspray, for example, you would want to be very confident your treatment wasn't likely to kill anyone, say 99.99%, but you'd be perfectly fine with a 75% confidence interval that your hairspray makes hair stay straight.

In general, confidence intervals should be used in such a fashion that you're comfortable with the uncertainty, but also not so strict they lower the power of your study into irrelevance. A 90% confidence interval means when repeating the sampling you would expect that one time in ten intervals generate will not include the true value. Based on what you're researching, is that acceptable? On the other hand, if you prefer a 99% confidence interval, is your sample size sufficient that your interval isn't going to be uselessly large? (Hopefully you're deciding the CI level before doing the study, right?)

In my experience (in the social sciences) and from what I've seen of my wife's (in the biological sciences), while there are CI/significance sort-of-standards in various fields and various specific cases, it's not uncommon for the majority of debate over a topic be whether you appropriately set your CI interval or significance level. I've been in meetings where a statistician patiently explained to a client that while they may like a 99% two sided confidence interval, for their data to ever show significance they would have to increase their sample tenfold; and I've been in meetings where clients ask why none of their data shows a significant difference, where we patiently explain to them it's because they chose a high interval - or the reverse, everything is significant because a lower interval was requested. When you publish a paper, it's not uncommon for three reviewers to have three different opinions of your CI level, if it's not on the high end for your discipline.

What I suggest is to read some of the major papers in your field (as close to your specific topic as possible) and see what they use; combine that with your comfort level and sample size; and then be prepared to defend what you choose with that information at hand. Unless you're in a field with very strict rules - clinical trials I suspect are the only ones that are really that strict, at least from what I've seen - you'll not get anything better. (And if there are strict rules, I'd expect the major papers in your field to follow it!)

Improve this answer
$\endgroup$
5
  • 5
    $\begingroup$ There are thousands of hair sprays marketed. I imagine that we would prefer that none of them killed people. That $\alpha$ seems too lenient. ;) $\endgroup$
    – Alexis
    Jan 7, 2015 at 18:29
  • 3
    $\begingroup$ @Alexis Unfortunately, for every few thousand users, one of them is likely to forget never to use a lighter while spraying their hair... $\endgroup$
    – Joe
    Jan 7, 2015 at 18:31
  • 2
    $\begingroup$ "A 90% confidence interval means one time in ten you'll find an outlier." This is downright wrong, unless I'm misreading you $\endgroup$
    – shadowtalker
    Jan 7, 2015 at 21:56
  • 3
    $\begingroup$ 90% CI means that 90% of the time, the population mean is within the confidence interval, and 10% it is outside (on one side or the other) of the interval. Thus 1 time out of 10, your finding does not include the true mean. Perhaps 'outlier' is the wrong word (although CIs are often (mis)used for that purpose.) $\endgroup$
    – Joe
    Jan 7, 2015 at 22:15
  • $\begingroup$ @Joe, I realize this is an old comment section, but this is wrong. You can have a CI of any level of 'confidence' that never includes the true value. See here: stats.stackexchange.com/a/26457/176202 $\endgroup$
    – Frans Rodenburg
    May 14, 2019 at 3:12
21
$\begingroup$

Choosing a confidence interval range is a subjective decision. You could choose literally any confidence interval: 50%, 90%, 99,999%... etc. It is about how much confidence do you want to have. Probably the most commonly used are 95% CI.

As about interpretation and the link you provided... These kinds of interpretations are oversimplifications. Correlation is a good example, because in different contexts different values could be considered as "strong" or "weak" correlation, take a look at some random example from the web:

  • I once asked a chemist who was calibrating a laboratory instrument to a standard what value of the correlation coefficient she was looking for. “0.9 is too low. You need at least 0.98 or 0.99.” She got the number from a government guidance document.
  • I once asked an engineer who was conducting a regression analysis of a treatment process what value of the correlation coefficient he was looking for. “Anything between 0.6 and 0.8 is acceptable.” His college professor told him this.
  • I once asked a biologist who was conducting an ANOVA of the size of field mice living in contaminated versus pristine soils what value of the correlation coefficient he was looking for. He didn’t know, but his cutoff was 0.2 based on the smallest size difference his model could detect with the number of samples he had.

So sorry, but there are no shortcuts...

To get a better feeling what Confidence Intervals are you could read more on them e.g. here, here, or here.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ Nice quote. +1. $\endgroup$
    – amoeba
    Jan 7, 2015 at 11:04
  • $\begingroup$ What you say about correlations descriptions is correct. It is entirely field related. The descriptions in the link is for social sciences. I suppose a description for confidence interval would be field dependent too. $\endgroup$
    – Oliver Moran
    Jan 7, 2015 at 11:07
10
$\begingroup$

Although, generally the confidence levels are left to the discretion of the analyst, there are cases when they are set by laws and regulations. I'll give you two examples.

In banking supervision you must use 99% confidence level when computing certain risks, see p.2 in this Basel regulation.

FDA may instruct to use certain confidence levels for drug and device testing in their statistical methodologies.

Overall, it's a good practice to consult the expert in your field to find out what are the accepted practices and regulations concerning confidence levels.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.