How is the 95% in 2 standard deviations above different than 95% confidence interval?

Question

Is there a difference between the two uses of 95% in:

• mean + 2 standard deviations (statistically significant)
• confidence interval

Answer 1

They are different, but connected.

In a 95% significance test, you reject the null hypothesis if your test statistic is one of the 5% most extreme values that could be seen, assuming the null hypothesis to be true. The regions corresponding to the extreme values are defined in advance and are fixed, i.e. they don't depend on the data you collect. In the diagram below, with a standard normally distributed test statistic $z$, this corresponds to $z$ being to the left of the leftmost red line or to the right of the rightmost red line, i.e. more than 2 standard deviations away from the mean (0). Notice that the area of under the curve represents probability, and the area under the curve of the most extreme regions is (approximately) 5%.

Confidence intervals are random intervals: they depend on the data you collect, so you don't know what the confidence interval is until after you've collected the data. Confidence intervals are defined so that if you run enough experiments, the percentage of times that the interval contains the actual mean of the underlying population of data will be 95% (or whatever confidence level you're using).

The connection between significance tests and confidence intervals is that if your confidence interval does not contain the value of the mean that you're using in your null hypothesis (usually 0), then this is equivalent to rejecting the null hypothesis at the same confidence level. See the diagram below, where our confidence interval lies to the right of 0, and the null hypothesis would be rejected.

Formally, you can derive confidence intervals from significance tests using something called the Test Inversion Lemma.

Also, note that the above assumes normally distribution. For your 2 standard deviations to correspond to 95%, you are assuming normally distributed data (a bell curve, as in the diagrams above). Significance test, and confidence intervals, can work on data regardless of distribution, although normally distributed data is the most important case.