How to report: large sample sizes (10000+), significant small difference (1%)

It is known that with large sample sizes, even a small difference could be statistically significant although it may not be practically meaningful. For my current analysis, the difference in the proportions of drug adherence between two cohorts was only 1%. A simple test would show that it was statistically insignificant. However, after throwing other covariates into a logistic regression, the estimate for the cohort variable appeared to be marginally significant with p-value 0.03.

I am just seeking advice on how I should report on such a result? My thoughts are that:

1. Just report on the naive result from the chi-square test.

2. Report on the 3 way tables for cohorts, adherence and a covariate of interest.

3. Logistic regression on adherence with all covariates of interest for each cohort separately, and graph the odds ratio and CI for two cohort side by side.

4. Report on the actual result from logistic regression with cohort variable, and put some explanations on why the result is not meaningful.

1 Answer

Ideally and theoretically, you should decide on the test before seeing any data, and then report only on that test.

In your situation, I would say that you should report the test that makes sense based on what you know about your data generating process. It makes no sense to report on a simple $\chi^2$ test that does not account for covariates you know are relevant. If such covariates exist and are included in the logistic regression, then that is the model you should report.

Of course, this does not hold if the logistic regression was the result of "fishing for significance" by trying different models until something significant came out. But I assume you would not be asking here if you had done that.

And it's a great idea to discuss that your result, while statistically significant, may not be practically meaningful.