What should everyone know about Confidence Interval?

I know this could be considered naive. But the fact is that I'm a computer Science student who have done introductory course in data analysis (Coursera) and a Econometrics I course at my University. While I understood the concept of CI, still I feel that there are lot of missing elements in my interpretation. So I searched for what everyone should know about CI and couldn't find one. I feel that there should be one for the sake of Internet people. Maybe, this community can help the cause!

Update:

For @NickCox's comment : I'm uncertain about Why we can't say that the true value is present in the CI's ? Rather, we resort to the fact, that if we conduct the experiment 100 times, we will get the true value 95% times (if we take the pvalue is 0.05)

• I doubt that there is a consensus among statistical people on what everyone should know about confidence intervals. Views start with a common Bayesian opinion or argument that the idea is fundamentally flawed. The Wikipedia entry en.wikipedia.org/wiki/Confidence_interval is one version. On a more personal level all that we can see here is that you understand CI, but feel your understanding is incomplete. That needs to be made much more specific before anyone can address what you are missing. – Nick Cox Dec 18 '13 at 10:23
• I think the most important part of a CI is its derivation. Most people just memorize the formula and forget that it comes straight from first principles and high-school algebra. For example, nowhere in this mess does anyone slap down a two- or three-line derivation. That would have (hopefully) ended the whole conversation. Yes, it generalizes, but that's not where the meaning is. – shadowtalker Dec 18 '13 at 10:32
• @NickCox True, agreed. That is one of the reasons , I was interested in the answer to this everything type of question. – Learner Dec 18 '13 at 10:36
• @ssdecontrol Feel free to add your own answer to that thread if you think something is missing. – Nick Cox Dec 18 '13 at 10:36
• Response to Update: The key point remains that we don't know the true value. If we did, we would not need confidence intervals at all. All we can do is estimate it, and in principle we can do that repeatedly. – Nick Cox Dec 18 '13 at 11:43

The most important thing to understand about a X% confidence interval is that the following statement is false:

There is a X% chance that the CI contains the "true" value.

Instead, it means that if you duplicated your experiment an infinite number of times, 95% of the time your estimated 95% CI will contain the true value. If you can get your head around that, you'll be ahead of many people.

You can certainly learn a lot more by searching this site for "confidence interval", including critiques of their use. For example:

What does a confidence interval (vs. a credible interval) actually express?

• Great point. This again is one of the facts that that our Professor requested us to never forget! Maybe, the name was misleading! – Learner Dec 18 '13 at 10:39
• If I remember correctly, it is often more correct to say that the CI covers the true value AT LEAST 95% of the times the experiment is repeated. That is, the exact CI often cannot be derived. – tomka Dec 18 '13 at 11:41

That it isn't a Bayesian credible interval (i.e. from a frequentist point of view, the probability that the true value of the statistic lies within a 95% confidence interval is not 0.95, but one or zero). The idea of a confidence interval is not fundamentally flawed (from a Bayesian perspective), it is just an answer to a different question (see my answer to this related question).

The confidence interval is the answer to the request: "Give me an interval that will bracket the true value of the parameter in 100p% of the instances of an experiment that is repeated a large number of times." The credible interval is an answer to the request: "Give me an interval that brackets the true value with probability p given the particular sample I've actually observed."

The frequentist confidence interval fundamentally cannot answer the second question. If you ask a Bayesian the first question, the answer you get will be very likely to be the same as the frequentist confidence interval (for some appropriate choice of prior).

The key think to remember is to interpret a confidence interval as a confidence interval, not as a credible interval.

It's also important to understand how parameter values inside the confidence interval differ from those outside, which is not something that follows from the definition of a confidence interval but from the particular method of its construction. For example, if you construct a confidence interval based on likelihood cut-offs the parameter values inside have higher likelihood than those outside. See Are all values within a 95% confidence interval equally likely?. To my mind such considerations are what put the "confidence" into "confidence interval"; taking only coverage into account allows the construction of valid confidence intervals that defy common sense. See Confidence interval for $\mathrm{Uniform}(\theta, \theta+a)$ & Confidence Interval for variance given one observation.