Confidence Interval “barely” includes the null, interpret as slightly increased risk of mortality?

Question

I was reading this paper:

Peralta, G., Sánchez, M.B., Roiz, M.P. et al. Diabetes does not affect outcome in patients with Enterobacteriaceae bacteremia. BMC Infect Dis 9, 94 (2009). https://doi.org/10.1186/1471-2334-9-94

These authors in the discussion section mention a similar past study, where they say that "this group found a slightly higher risk of late death in diabetic patients".

However when I looked at the study they are citing:

Reimar W. Thomsen, Heidi H. Hundborg, Hans-Henrik Lervang, Søren P. Johnsen, Henrik C. Schønheyder, Henrik T. Sørensen, Diabetes Mellitus as a Risk and Prognostic Factor for Community-Acquired Bacteremia Due to Enterobacteria: A 10-Year, Population-Based Study among Adults, Clinical Infectious Diseases, Volume 40, Issue 4, 15 February 2005, Pages 628–631, https://doi.org/10.1086/427699

The results section indicated that the results were in fact not significant (you might need to look at the PDF version to see these numbers):

At 30 days, the mortality was 17.3% for diabetic patients and 13.4% for nondiabetic patients; after 90 days, it was 23.6% and 19.5%, respectively. After 30 days, the adjusted mortality rate ratio for diabetic patients was 1.3 (95% CI, 0.9–1.8), and after 90 days it was 1.2 (95% CI, 0.9–1.6). Stratification for focus of infection did not materially affect the mortality estimates. When focus of infection was included in the analyses, the mortality rate ratio for diabetic patients was virtually unchanged: 1.4 (95% CI, 1.0–2.0) after 30 days and 1.3 (95% CI, 0.9–1.7) after 90 days.

All of these confidence intervals appear to (barely) cross the null of 1. I am wondering why Peralta et al. appears to interpret this as a "slightly higher risk"? Is it because the results are only just barely including the null? Can it be appropriate to interpret confidence intervals in this way, that even though they are not "statistically significant" they can still perhaps suggest an elevated risk since they are almost entirely above the null?

Answer 1

To some extent, I think they are mixing up statistical and practical significance. What I think they're doing, though, is saying that they cannot reject at $\alpha=0.05$ but can reject at $0.10$. Even though they do not show the $90\%$ confidence intervals, I have a hunch that the lower endpoints exceed $1$.

(Is my head going to explode before I finish writing this answer? No? Good!)

The p-value tells you how surprising of a result you have. It does not tell you by how much your observations differ from what your null value, since a small deviation from the null value combined with a large sample size could be very strong evidence that the null hypothesis is wrong, even if just barely.$^{\dagger}$ That is the realm of effect size, and effect size would quantify the extent to which one group has a higher mortality rate.

Doubling your chance of dying (highest upper endpoint is $2$) sounds like a big deal to me. However, perhaps it isn't, and that is only "slightly higher risk" like the article mentions.

$^{\dagger}$Think about getting a few more heads than tails when you flip a coin a trillion times. You might be convinced that the coin has a bias towards heads because it lands that way 50.005% of the time, but that might not be enough of a deviation from perfect 50/50 for you to care. This would have statistical significance (convincing you that the true proportion is not 50/50) but insignificant practical significance (not different enough to be interesting).

Answer 2

A core issue which the ASA has tried to address in recent years (see Wasserstein, Schirm & Lazar 2019) is the misinterpretation of "statistical significance." The same issue arises here. When an estimated effect is "not statistically significant"--or here, where the confidence interval includes the null--this does NOT mean that the parameter value is null in the population. The best estimate of a parameter is the value of an unbiased estimator of that parameter. The authors reported that their best estimate put diabetes patients at slightly higher risk. The p-value / confidence interval shows a substantial degree of uncertainty around the estimate, but that does not support favoring any other value (like the null) over the best estimate.