Statistical Significance in Table Ones

Question

I have a question about statistical and clinical significance.

In a non-clinical trials setting (e.g. observational), if the sample size is large for both groups (assume 2 groups for simplicity) enough many covariates/characteristics in a Table One (usually called the Table of Patient Characteristics) would turn out to be statistically significant (P-value < 0.05). As we know, as N -> infinity we almost always reject the null.

For example, if we have a sample of 100,000 and the mean age for the treatment group is 71 and the mean age for the control group is 73. The underlying T-test in a Table One (R function tableone for example) would yield a P-value < 0.05. In other words, this difference in mean age is statistically significant largely due to having a large sample size. However, many clinicians won't find this difference meaningful.

Now, say we have a situation like this AND in addition to that, the clinicians don't know what the minimal clinically significant difference in mean age is. Is there a common 'threshold' that we statisticians can use to say which covariates/characteristics between the 2 groups are 'clinically significantly different'? I've looked into the MCID (minimal clinically significant difference) but it seems like the use of it is limited to variables with some measurement error.

Also, some argue that if the difference is > 0.5 * the SD it can be considered clinically significant, but I'm not sure how valid this is.

Please help and thank you!

Answer 1

Covariate [differences] in Table One are not statistically significant because they are not prespecified hypotheses. And nobody would take you seriously for prespecifying a hypothesis of differences in, say, sex and age when running an RCT. Adjust for covariates in the final model if you are worried about imbalance and the imapct.

If the hypothesis is not prespecified there is certainly no reason to speak of clinically meaningful differences. My suggestion is to delete all $p$-values from table 1 and never report them.

If the study is randomized, you actually still expect the (ITT) differences to have $p < 0.05$ only 1 in 20 times even when the null is perfectly true. Blocking randomization can improve balance (and efficiency).