0
$\begingroup$

I am unclear about the degree of overlap in the information contained by the CI as opposed to that contained in the p-value and effect size of that same statistic/estimate.

I know of course that if a 95% CI does not include the zero value of a particular effect, then it can be inferred that said effect is statistically significant, in the same way that this would be inferred from a p<0.05. And, of course, that CIs can be calculated from p-values, and vice-versa.

But what is unclear to me is:

1) if the CI is reported, is there any need at all to still provide the p-value? In other words, does it give any extra information that the CI does not already give?

2) if I want to describe, say, the difference in means between two groups, does it still make sense to provide a measure of effect size such as Cohen's d if I have already provided the CI of the sample point-estimate?

Reading the paper "Confidence Interval or P-Value?" by du Prel et al. (2009) has left me still in doubt unfortunately, so further reading that addresses my question would be appreciated in an answer - thanks in advance.

I should also mentin that this question follows on from an earlier one to which I never obtained a satisfactory answer, just as searching previous questions on here was not entirely enlightening.

$\endgroup$
1
$\begingroup$

While information such as differences in means and p-values can often be back-calculated from a confidence interval (especially if it is of the form $\hat{\delta}\pm c_{1-\alpha/2} \text{SE}(\hat{\delta})$) potentially after some transformation), it is not easy for people to do this in their head. So, it is better to do these calculatons for them. Particularly the difference in means should really be provided in my opinion, while it is in many fields expected to also be provided p-values (even if people usually mis-interpret them).

Talking for my specific field, for controlled clinical trials the standard set of information I would always expect is difference in means (or some other point estimate), confidence interval (or something else such as a credible interval) and p-value (especially if it is multiplicity adjusted - multiplicity adjusted confidence intervals are rarely reported and hard to obtain). In addition, I would normally expect to see something like adjusted least-squares means (or some similar measure) for each group with confidence interval or standard error, as well as information on the number of patients in each group. This covers a diverse set of needs: an idea of the effect size, an idea of the uncertainty around it, what people are used to seeing, what you would need to include the trial into a meta-analysis, what you would need to construct a historical prior for a new trial, what you need to get an idea of the variability of the outcome (useful for sample size calculations for new trials) etc.

$\endgroup$
1
$\begingroup$

I will share what I tell my students: our job in conveying statistical information to our "clients" is to quantify what was observed and to provide some measure of how this evidence can be assessed. I would argue that the sample statistics and effect sizes address the former, and confidence intervals (CI) or $P$-values address the latter. Thus, I would suggest reporting both the CI and the effect size, such as $d$ or $R^2$. However, reporting both the CI and the $P$-value would be redundant.

Regarding the 2nd query, the size of the CI depends on $n$, and thus can be misleading. That is, you can have two CIs with the same distance from zero and the same $d$. One CI could be very large and the other very small. Thus, you do provide additional information by reporting the effect size, as well.

Lastly, regarding additional readings, I can't think of any right now. But, there is a lot that shows up in most modern statistics/research-design textbooks (particularly with applied focus, like for social science researchers).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.