6
$\begingroup$

A well-known rule of thumb is that, for an estimate to be significantly different from 0, its 95% CI has to not include 0, i.e. the lower and the upper bounds of the CI have to be either both positive or both negative.

However, I know that for certain measures of effect size, the CI does not abide by this rule. For instance, for partial eta squared, the lower bound of its CI is by definition >=0 (Kline, 2004). Thus, anyone who looks at the CI of the effect size and finds it does not contain 0, might mistakenly believe that the reported effect is significant, whereas this will not necessarily be the case.

Thus, how are those two things reconciled, and wherein does my misunderstanding lie?


Reference: Kline RB (2004) Beyond significance testing. Washington, DC: American Psychological Association

$\endgroup$
1
  • 2
    $\begingroup$ I think the essence of the answer here is that some effect size statistics are directional --- that is, that they can be positive, negative, or 0. While other effect size statistics are non-directional --- that is, they are always 0 or positive. Inference with the confidence intervals for these statistics is challenging. I wrote a long answer unpacking this, as I don't think the currently accepted answer really addresses the question as posed. $\endgroup$ Nov 21, 2019 at 17:20

4 Answers 4

4
$\begingroup$

One salient point in this question is that it refers to effect size statistics. The fact that these statistics can be either directional or non-directional, I think is the essence of the answer to the question.

Some effect size statistics are directional †. That is, they can contain positive, negative, or zero values, with a positive value indicating a positive correlation, or, usually, the values for the first group being larger than the second. These include r, Spearman's rho, Kendall's tau, phi for 2 x 2 contingency tables, Cohen's d for the means of two groups, and Cliff's delta for the ranks of two groups.

In these cases, it's a fair interpretation that if the confidence interval does not include zero, then the effect size statistic is "significant". An exception here is that if the statistic is reported as the absolute value ‡ --- that is, always positive --- confidence intervals by bootstrap may not reflect this property. As a final note here, since there are different ways to compute confidence intervals (e.g. by formula or by bootstrap), this conclusion may not always match exactly the related hypothesis test. (That is, say, a 95% CI for Cohen's d may not match exactly the results from a t test).

There are other effect size statistics that are always positive. Examples include r-squared, eta-squared, partial eta-squared, Cramer's v that is used for contingency tables larger than 2 x 2, and the epsilon-squared that is sometimes used for Kruskal-Wallis. Often these are used in situations where it is not easy to convey the effect size in a negative-or-zero-or-positive framework. For example, while we can use phi in a 2 x 2 contingency table, and it makes sense to look at the correlation as positive or negative, in larger tables there may not be a clear way to think about the correlation as positive or negative.

For these statistics, because they are always greater than or equal to zero, an accurate confidence interval would never include values less than zero. Some calculations for confidence intervals may include zero, which would be a potential indicator of "statistically zero effect", but usually calculations won't be so precise. In the case of determining confidence intervals by bootstrap, the range would be unlikely to contain a zero by a method like percentile, although if the confidence interval were determined by normal bootstrap, the interval could cross zero. If this latter method were appropriate for a specific statistic of concern, this approach may perhaps be viable for inference.

One final note, there's an interesting case of Vargha and Delaney's A, which is directional, but where 0.50 indicates no effect.


† I have seen this term used this way for effect size statistics, but I'm not sure it's meaning is universal in this context.

‡ For example, I've seen R functions that report the absolute value of certain effect size statistics.

$\endgroup$
3
  • $\begingroup$ THat indeed addresses my question perfectly - thank you! $\endgroup$
    – z8080
    Nov 22, 2019 at 15:28
  • $\begingroup$ "One final note, there's an interesting case of Vargha and Delaney's A, which is directional, but where 0.50 indicates no effect." this counters the entire point that you make in this answer. Yes, often situations where the zero relates to 'no effect' are cases that are directional. But this is not a principle and Vargha and Delaney's A that you mention is a counter example. Other cases are tests for some theoretical prediction different from zero. $\endgroup$ Dec 14, 2023 at 18:30
  • $\begingroup$ @SextusEmpiricus , Another common effect size statistic that is directional but doesn't have 0 as a "no effect" point is odds ratio. But I don't think these are counter examples to the question or my response. For either, we could just change the statement to, "If the CI does not contain 1..." or "contain 0.50". ... The issue is really those statistics (like r-squared), whose CI will never never really cross zero, and --- depending how the confidence interval is calculated --- may practically rarely contain 0, even if there is statistically no effect... $\endgroup$ Dec 16, 2023 at 17:31
7
$\begingroup$

It depends on what kind of test you’re doing.

For a difference between the values from two distributions, a nonzero difference indicates an effect. This would cover the usual testing of mean differences with $H_0: \mu_0-\mu_1=0$.

However, with testing variances, it is more common to consider the ratio of variances. In this case, the null would be $H_0: \sigma^2_0/\sigma^2_1 =1$. Consequently, it is desired for the confidence interval of that ratio not to include 1. Zero doesn’t come into the mix. A ratio of zero means that the top variance is zero.

So anyone seeing a confidence interval for a ratio of variances of $(0.8, 1.9)$ and calling the effect significant has made a mistake.

The theory behind this is that a confidence interval is an inversion of a hypothesis test. The confidence interval partitions the space of possible values into what would and would not be rejected by an $\alpha$-level test.

Casella and Berger get into this when they talk about interval estimation. Their unfortunate terminology calls these partitions the “rejection region” and “acceptance region” (even though we do not quite “accept” a null hypothesis).

(I have seen on here that some exotic confidence intervals need not obey this rule, and I will allow someone else to address such details.)

All of this assumes a two-sample comparison. In the one-sample case, we still have the same thinking that we want to check a confidence interval for some surmised value, such as $\mu=\mu_0$ where $\mu_0$ need not be zero.

(The two-sample case could use a null of something other than equality, such as $H_0: \mu_0-\mu_1=6$. Then the interesting question is if the confidence interval contains 6.)

$\endgroup$
5
2
$\begingroup$

Besides the obvious cases where the null hypothesis does not refer to zero - at least not without a transformation - such as for an odds ratio, hazard ratio, rate ratio (these first 3 of course are just a matter of a suitable transformation) etc., as well as non-inferiority testing and equivalence testing (but here, it's again mostly following the principle of "does the CI lie entirely within the alternative"?), there's also cases where it's really hard to find "good" confidence intervals that match up with tests.

One arguable example of this is the case of comparing two binomial proportions, where a test of the null hypothesis can obviously use SE based on a variance estimate that is only valid under the null hypothesis of equal proportions, while for a confidence interval you'd want a variance estimator that is also valid under the alternative hypothesis. As a result you can have a mismatch between test and confidence interval.

Another scenario where it is not easy to get really useful confidence intervals that match up with test decisions is when you adjust for multiplicity using complex multiple testing procedures (e.g. some complicated closed testing procedure), although the CIs will often end at the null hypothesis value when the null hypothesis is rejected (but may not move away from there even as more and more evidence becomes available).

$\endgroup$
4
  • $\begingroup$ I do not find the argument for that example strong. Sure, you can break the duality between confidence intervals and p-values. But if that happens then it is because there are multiple types of intervals and multiple types of hypothesis tests and non-matching cases are compared with each other. For every confidence interval there is a matching hypothesis test. Confidence interval / p-value duality: don't they use different distributions? $\endgroup$ Dec 14, 2023 at 18:21
  • $\begingroup$ "while for a confidence interval you'd want a variance estimator that is also valid under the alternative hypothesis" For a confidence interval you estimate the variance seperately for each value inside and outside the interval. But, for the specific value that coincides with a null effect, the variance is computed just like a test for a null hypothesis. $\endgroup$ Dec 14, 2023 at 18:24
  • $\begingroup$ @SextusEmpiricus Sure, you can find a matching test for every CI, but you cannot find a matching valid CI for every test. $\endgroup$
    – Björn
    Dec 15, 2023 at 13:51
  • $\begingroup$ Which of the two situations is the topic of this question? $\endgroup$ Dec 15, 2023 at 14:14
1
$\begingroup$

"if the 95% CI does not contain 0, then the effect is significant"

Every method to compute a confidence interval can be converted into a method to compute a confidence distribution and associated p-values for hypothesis tests.

Confidence interval / p-value duality: don't they use different distributions?

Can we reject a null hypothesis with confidence intervals produced via sampling rather than the null hypothesis?

Using that duality, if the $\alpha \%$ CI for a parameter doesn't contain 0, then we can say that a p-value for the associated hypothesis test for the parameter being equal to 0 is below $100 \% -\alpha$.

The remaining issue in the question is whether "the effect being significantly different from 0" coincides with the phrase "the effect is significant". The latter term is not very clear. The absence of an effect does not need to coincide with a parameter being equal to zero.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.