6
$\begingroup$

So, the higher the confidence interval the lower the false positive rate, but the false negative rate will increase lowering the recall.

Is it possible to determine which confidence interval is better/more significant? 95% or 80%?

Also why is the lower limit for confidence interval is generally 80%?

Why is it not 70% or 60%?

$\endgroup$
3
  • 4
    $\begingroup$ You should choose your level of significance $\alpha$ before doing the test. If $\alpha$ is smaller then it is less likely to get a significant result from the test (especially if the null hypothesis is correct - desirable) and the associated $1-\alpha$ confidence interval is wider making it more likely the confidence interval will cover the true value. The power of the test to reject the null hypothesis when a given alternative hypothesis is correct will also get smaller, though is also affected by other factors such as design of the test and the sample size. $\endgroup$
    – Henry
    Commented Aug 18, 2023 at 7:10
  • 1
    $\begingroup$ Welcome to cv, Ankita! We would not call a confidence interval "significant" - did you read this somewhere? The 95% or 80% or whatever are "confidence levels". $\endgroup$
    – Ute
    Commented Aug 18, 2023 at 7:10
  • $\begingroup$ Thanks for correcting @Ute .I actually have heard a lot of people using it even in Professional fields so was not sure if is correct.Will definitely keep in mind going forward! $\endgroup$
    – Ankita
    Commented Aug 22, 2023 at 6:25

3 Answers 3

15
$\begingroup$

The language of statistical "significance" is not directly applicable to a confidence interval, though intervals are often related to similar hypothesis tests. If you were to compare a confidence interval to a corresponding hypothesis test you would find that the "confidence level" $1-\alpha$ and the "significance level" $\alpha$ are in fact negatively related --- i.e., a higher confidence level corresponds to a lower significance level. However, saying that a result in a hypothesis test is "more significant" usually means that it has a lower p-value, which is determined by the data, not the chosen significance level. For all these reasons, I strongly recommend that you avoid bringing the terminology of "significance" into discussion of a confidence interval. (It is bad enough in its proper context!)

Just as with significance levels in a hypothesis test, there are no sacrosanct values for the confidence level of a confidence interval. The choice of an appropriate confidence level depends on context, and it always involves a trade-off between desired levels of confidence and accuracy. For a fixed set of data and a fixed procedure for the confidence interval, if you choose a higher confidence level then you will get a wider (less accurate) interval, and vice versa. You can think of a higher confidence level as a more conservative inference, insofar as you are willing to be less accurate in order to be highly confident that the parameter of interest falls within the interval. Contrarily, you can think of a lower confidence interval as a less conservative inference, since you will get better accuracy for the interval but at the cost of having lower confidence that the parameter of interest falls within the interval.

There is no inherent problem with choosing a confidence interval at 60% or 70% confidence level, though obviously that is quite a low level of confidence and so it is a risky inference. Typically people choose higher confidence levels because they want to have a high level of confidence that the unknown value of interest falls within the computed interval. Personally, I tend to be quite conservative in my statistical inferences, so unless there are good contextual reasons to the contrary, if I have a reasonable amount of data I will usually opt for a 99% confidence interval (or perhaps an even higher confidence level).

$\endgroup$
5
  • 1
    $\begingroup$ I just want to add a note of caution about using high (e.g. 99%) confidence levels with correspondingly wider confidence intervals: as @PeterFlom states in his answer this isn't cost-free. In some sense, and particularly for a layman, a 99% CI can be misleadingly wide, since most of the distribution is contained in the centre rather than the tails. A Bayesian would say it's more probable the true value is near the centre of the CI than the tails, but even sticking to a frequentist framework, we can say a 75% or 80% CI is much narrower than 99% $\endgroup$
    – Silverfish
    Commented Aug 19, 2023 at 3:30
  • 1
    $\begingroup$ As Ben explains, there is an equivalence between the null hypothesis statistical testing (NHST) framework and CIs: if you view a CI as a hypothesis test in disguise, and have a conservative approach to NHST, this is grounds for sticking to a high level and wide CI. But to a layman this may seem to overstate the uncertainty. I see an 80% or 90% CI would be regarded by many here as insufficiently conservative, but it's worth contrasting this to standardised words of estimative probability. $\endgroup$
    – Silverfish
    Commented Aug 19, 2023 at 3:37
  • 2
    $\begingroup$ The PHIA Probability Yardstick in UK public sector analysis (especially intelligence/policing) rates 80% as "highly likely". The IPCC communicates 90% as "very likely". If you simply wish to express what range of values for an estimated parameter is "likely", you may find a 75%, 80% or 90% CI communicates an appropriate level of uncertainty. But as Ben says, when thought about in the NHST framework, this usually won't correspond to a meaningful significance level, so if e.g. zero doesn't lie in the interval, we shouldn't draw an inferences like "there's strong evidence the effect is non-zero" $\endgroup$
    – Silverfish
    Commented Aug 19, 2023 at 3:46
  • $\begingroup$ Hi @Ben ,thanks for the detailed answer.Point noted for the usage(or no usage) of word " Significance".but can we really say that the lesser confidence level will lead to more accurate results?SHouldn't it be higher confidence level leading to more accurate however lesser confidence level leading to more precise results? $\endgroup$
    – Ankita
    Commented Aug 22, 2023 at 8:20
  • $\begingroup$ @Ankita: Re CIs, for a fixed set of data and a fixed method, there is a trade-off --- you cannot have both more confidence and more accuracy for your inference (for that you would require more data). If you want to increase confidence you have to settle for less accuracy; if you want more accuracy you have to settle for less confidence. If you'd like, try computing CIs at different confidence levels on a fixed set of data and see what these look like; I think you will find that the 99% CI is wider (less accurate) than the 95% CI, which is wider (less accurate) than the 90% CI, and so on. $\endgroup$
    – Ben
    Commented Aug 22, 2023 at 9:54
9
$\begingroup$

@Ben gave an excellent answer (+1) I'm just adding to it.

You have to decide which error is worse and by how much. In psychology, where I did most of my work, there's a strong convention for power of 0.8 and alpha of 0.05. This says that a type I error is 4 times worse than a type II error. But that's not always true or even remotely reasonable. Fisher himself said that people should vary their significance levels.

Suppose you come up with a new treatment for a disease that is quickly terminal. Then a type I error means you give a useless treatment to dying people, but a type II error means you fail to save lives that you could have saved.

Now suppose you develop a new treatment for teenage acne. It is much more expensive than current medications and has some nasty side effects. Now a type I error means you cost people money and give them side effects for no reason, but a type II error means a kid has acne for a little bit longer.

Another example of the truth of Sir David Cox's quotation:

There are no routine statistical questions, only questionable statistical routines. He uses statistics the way a drunken man uses a lamp post: More for support than illumination.

$\endgroup$
3
  • 1
    $\begingroup$ Thanks so much this is great.Especially the quotation. Also one question here:in the first example type I error means we are treating people who doesn't have the illness right? $\endgroup$
    – Ankita
    Commented Aug 22, 2023 at 8:47
  • $\begingroup$ No. A type 1 error means you think the drug does some thing, but it doesn't. So, sick people get a useless drug. The question of whether people actually have the disease is a different issue altogether. $\endgroup$
    – Peter Flom
    Commented Aug 22, 2023 at 10:06
  • $\begingroup$ oh, I was thinking about the method to find out the terminal illness is the product here,but got it now,Thanks! $\endgroup$
    – Ankita
    Commented Sep 6, 2023 at 18:02
4
$\begingroup$

The term "significance" should optimally be used referring to test results together with a level indication, such as "the data provide a significant indication against the null hypothesis at level 0.05". Using it with p-values such as "p=0.005 is more significant than p=0.04" is still correct, but somewhat harder to process for non-statisticians. Generally significance always refers to data being incompatible, according to a statistical test, with the data. Otherwise in the interest of clear communication better don't use the word "significant" referring to statistical results.

In particular, the term does not apply to confidence intervals, despite the mathematical relation between confidence levels and significance levels as explained by @Ben. A confidence interval does not regard a single null hypothesis, which is what "significance" is about.

There are no better or worse confidence/significance levels, as there is always a trade-off. The larger the confidence level, the bigger the confidence interval becomes, which means it indicates less precisely where the parameter is, but the smaller the "error probability", i.e., the probability of having the true value not captured in the interval. We want a small interval and a small error probability, so we need to decide how to balance these.

The only thing that can be said is that we want to have a large probability to not miss the true parameter, or, for a test, a small probability to reject a true H0, so confidence levels should be "large" and significance levels "small", but "small" can mean 0.05, 0.01, even 0.001 or 0.1, and values such as 0.019 should only be avoided because they smell of "the author just picked what they needed", so better use widely applied standards, but there isn't anything intrinsically wrong with them.

80% as confidence level is already quite low in my view, it means "1 in 5 cases we go wrong", which, so to say, "can happen all the time", so I wouldn't go that low or even lower, but mathematically also 80% or 65% confidence intervals are not invalid.

$\endgroup$
1
  • 1
    $\begingroup$ I left a comment under Ben's answer that's relevant here too: in standardised words of estimative probability, a CI with 65% or 80% coverage may well be deemed "likely" or "highly likely" to include the true value. Provided you don't mistake this for "strong" or "very strong" evidence in the NHST framework, being wrong 1 time in 5 can still give a useful impression of the uncertainty $\endgroup$
    – Silverfish
    Commented Aug 19, 2023 at 3:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.