2
$\begingroup$

Do confidence intervals (CIs) also include information on effect size, in addition to statistical significance - enough to warrant not reporting each of those separately?

My understanding is that CIs replace the need for p-values, in that if a statistic is significantly different from 0 e.g. at the 0.05 level, then the 95% CI for that statistic will not contain 0.

But my question is: if the CI is provided and can be taken as an interval estimate of effect size, then does this eliminate the need to also specify the actual effect size, insofar as (correct me if I'm wrong) the latter is merely the point-estimate of the former?

And isn't this just a case of estimation statistics seeking to replace null hypothesis significance testing [*] ??

To illustrate this in the context of specific analyses: 1) For an ANOVA for which a partial-eta-square statistic is chosen as the measure of effect size, then would the corresponding CI have to be given for this particular effect-size statistic, or for the reported effect itself (e.g. main or interaction effect)? 2) For a correlation for which Pearson's R is chosen as the measure of effect size, and if CIs are provided for R, is there any point to still reporting the (point estimate) of R itself, as long as it can be assumed that the interval-estimate (the CI) is symmetrical about this point?

I might be mixing&confusing several key concepts here, and I appreciate the patience of any answers that will try and clarify those different concepts.

[*] Cumming, Geoff (2012). Understanding The New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis. New York: Routledge.

$\endgroup$
  • $\begingroup$ The effect size is just another statistic like the mean. You can actually use the delta method to generate confidence intervals for the effect size. Typically the effect size can be backed out of the p-value, which is to say the effect size typically provides no new information just the same information in a different way. $\endgroup$ – Michael Naaman Aug 8 at 13:46
3
$\begingroup$

According to Wikipedia:

About 50 to 100 different measures of effect size are known.

This thread presents several takes on meanings of the term. Another thread, somewhat related to your question, discusses the different information conveyed by effect sizes and p-values (and thus by CI related to those p-values).

I can find the term "effect size" to be more confusing than clarifying. I do like the idea of trying to present a sense of the practical significance of a "statistically significant" finding, which is what those who report "effect sizes" typically are trying to do. But that well intentioned effort can readily go astray. As Jeromy Anglim's answer to What descriptive statistics are not effect sizes? puts it:

Any statistic that communicates the degree of relationships and is not especially contaminated by sample size is probably an effect size measure. Thus, Beta coefficients, R-square, covariance, raw mean differences between groups, and so on all capture the degree of effect. That said, I find that some researchers apply effect size measures somewhat blindly and forget that the broader aim is to give readers a sense of the degree of effect.

So focus on giving your audience "a sense of the degree of effect."

Point estimates are often most useful toward that end. Confidence intervals (CI) then represent just that--a type of measure of confidence in the point estimates.

Things get a bit more complicated when your measure of "effect size" is a scaled version of a raw effect size. Cohen's d comes to mind. The raw mean difference between two groups is already a type of effect size; Cohen's d scales that by a pooled estimate of the within-group standard deviation to provide a unitless effect size measure. Which gives a better "sense of the degree of effect"? The answer to that question can depend on the subject matter at hand. And if you want to provide Cohen's d as the effect size measure, one could argue that CI for that measure should also be provided.

Two more notes. First, several measures of effect size (e.g., Cohen's d and Pearson's r) can be directly related to p-values and CI if you know the sample size and certain assumptions are met. Second, you can't assume that CI are symmetric about the point estimate. In your example of Pearson's r, values are restricted to the interval [-1,1] and thus CI near the extremes can't be symmetric. CI for Pearson's r based on Fishers's transformation are symmetric in an hyperbolic arc-tangent transformation, but I don't think that presenting values of $\operatorname{artanh}(r)$ would give most readers a useful "sense of the degree of effect."

$\endgroup$
  • $\begingroup$ Thanks for your answer, it's a helpful start. I don't think I was hung up on terminology at all - quite to the contrary, I was trying to grasp a very broad&theoretical concept, that of effect sizes (however they might be conceptualised) estimated either by points or by intervals. $\endgroup$ – z8080 May 11 '18 at 9:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.