Whether to report confidence intervals of effect sizes such as $r$ and $\eta^2$?

I have calculated the bivariate linear correlation coefficient between $x$ and $y$ and it is $r= 0.45$; $p<001$.

I know $r$ is a measure of effect size.

Should I also be reporting the confidence interval in this case?

If yes, how can it be justified? i.e. why is it important to report it together with the effect size ($r$) - a bit of non-statistical explanation will certainly be very useful for me.

The same is for my linear regression analysis of $x$ and $y$. I am reporting the $F$ statistics, $p<0.05$ and $\eta^2$.

Should I also work out the confidence interval for the linear regression analysis?

The answer is almost always: report both. This way, your audience can decide on the interestingness and importance of your results, instead of just having to believe you. Confidence intervals are similarly always useful, because they give a neat indication of both effect size, and significance, in one. Even better if it's on a graph :)

It may be best to look at all the possible outcomes, remembering that a significance level ($\alpha$) is an arbitrary threshold, and you can choose anyone you like (higher ones are of course harder to defend).

1. Low p-value, low effect size: You've got a result, you're pretty sure it's not down to chance, but it doesn't really say anything interesting. An example might be a new drug, that significantly improves on an old one. But if the improvement is a only 2%, then your result might not means so much when weighed up against other factors (like extra costs or new side effects)

2. High p-value, high effect size: Looks like you're on to some thing, but you can't say for certain that it wasn't just the result of chance. For this drug, things look promising, but you DEFINITELY want to do some more testing, probably with a modified experiment that (hopefully) will reduce some of that ridiculous variability you're seeing.

3. High p-value, low effect size: Nothing interesting here. Go and design a better experiment.

4. Low p-value, large effect size: Win. You've got a big effect, and you're sure it's not down to chance. If it's a new drug, then you've got a good chance that your results will make a big impact, and it'll get pushed out to market quickly, even if it costs more, or if there are some side effects.

(Note: not pushing drugs here, I'm as mistrustful as the next paranoiac about big pharma, but drugs make for good statistical examples :)

• The meaning of the terms high and low significance have to be divined from your context. In addition, significance is generally categorical. I'm guessing you mean high and low p-values, but in that case you would be using p as a measure of evidence for the effect, not to indicate significance. It might be better to rewrite using "low p-value, high effect size", for example.
– John
Apr 27 '12 at 6:38
• thanks John, I've incorporated your suggestion. I'm not sure that I'm with you on "significance is categorical" though. Surely you can have two significant results, with one more significant than the other? Apr 27 '12 at 7:36
• A net search of "what is statistical significance" will yield references explaining that it is categorical. Typically you set a significance level, alpha. If a p-value is below that it is statistically significant. The level is something you set before hand to define the categorical distinction, not something the p-value inherently contains. You're thinking of significance as the amount of evidence for the effect, which is a different way to treat a p-value from significance and they can't be mixed together. (e.g. see Gigerenzer, "Mindless Statistics")
– John
Apr 27 '12 at 12:51
• Thanks, I just noticed I said p-value when I meant significance level. Will fix. I was trying to get at the fact that the significance level is arbitrary, and a p-value below a lower significance level is a more convincing result. I guess I see categorical significance as kind of a stupid concept (especially at $\alpha =0.05$).. I agree with you, I'm just not so good with the terminology :) ... Thanks for the Gigenrenzer reference. very useful. Apr 27 '12 at 13:17
• You should probably formulate this into a new question on here. You would need to provide some context because there is no standard wide and narrow. But to be really brief and vague, you have to decide how big an effect is important or not. A narrow CI would be smaller than that and a wide CI would be bigger.
– John
Apr 30 '12 at 0:57

Yes, you should calculate intervals for your statistics. The hard part is not justifying the interval presentation, it would be more difficult to justify not presenting it. It is very likely the exact effect size, mean, or whatever statistic you have calculated as a model for your data is not the true value. Calculating an interval reflects that A) you know that it's not the true value, and B) this is the area where you believe the true value to be. Intervals around the values you calculate allow one to make inferences beyond simple significance tests as well as indicating the power of the experiment.

Also, with a simple linear regression, as you have described, it is often best to report the beta coefficient, and a confidence interval. You could also report one for whatever effect size you select. If your predictor is continuous it's kinda hard to imagine what the CI looks like and it's best to plot it. Your stats package should be able to help with that. For example, in R you could get predict() to return CI values to plot.

• Thanks. SPSS gives partial eta for the GLM. I have converted that to eta squared because it is analogous to r. Apr 30 '12 at 0:19