1
$\begingroup$

A typical reality: a statistical journal asks me (I cannot refuse or won't get published) to summarize several baseline characteristics and compare them via test across 2 groups, G1 and G2, namely:

  • age at G1 and G2
  • gender % at G1 and G2
  • some clinical parameter at G1 and G2
  • % of co-morbidities at G1 and G2
  • ... 10 additional measures at G1 and G2

Neither of the above comparisons share the same data.

In other words, it's not like testing for pairwise differences: baseline-time1, baseline-time2, baseline-time3, time1-time2, time1-time3, time2-time3 and so on, where we naturally adjust the significance level (here using the Dunnett or Tukey, for instance).

It's just a set of tests for a set of very different outcomes.

So I get:

Sex:
    males/females: G1=xx%, G2=xx%
    diff=xx, 95% CI= [xx - xx]
    p-value = 0.xx
    effect size ABC = xx, small

Age:
    mean: G1=xx.x, G2=xx.x
    diff=xx.x, 95% CI= [xx.x - xx.x]
    p-value = 0.xx
    effect size ABC = xx, moderate

and so on.

The reported p-values and CIs are unadjusted here.

In my opinion this summary should be WITHOUT any statistical inference, as it just summarizes the "local" data from the experiment, but the journals demands me to add this, so I am going to add it. Let's not question that and focus on my problem.

Should I leave the p-values unadjusted, because every comparison is about something very different, no "p-catching", or adjust them for something. But for what?

Having 15 parameters, I'd need to adjust 0.05/15 = 0.003(3), assuming the simplest Bonferroni adjustment (I could also use the Holm more power, I guess).

But is this really necessary in THIS very case?

Please note, I'm aware of the fallacy saying that "one doesn't have to adjust for orthogonal hypotheses or planned comparisons". Here it's not an orthogonal contrast (like effect coding or Helmert), it's just a set of different questions.

The goal of the summary is not to find "at least rejected hypothesis in the experiment" (because the experiment has it's own primary objective), rather to describe the "homogeneity" of groups, to say: "OK, the two groups were homogeneous in terms of age and sex", which sets the context for further analyses.

Improve this question
$\endgroup$
1
  • $\begingroup$ If data are different for each comparison, I see no need for protection against false disclosure. It might be confusing to readers to adjust P-values as if multiple inferences were made for the same data. If these comparisons were not part of the study as planned, I would feel free to say so. $\endgroup$
    – BruceET
    Commented Apr 10, 2022 at 7:32

1 Answer 1

Reset to default
2
$\begingroup$

Welcome to SE!

I agree with you that p-values in such tables are rarely a good idea (but often requested by journals). Usually, the hypothesis tests presented for baseline characteristics are not adjusted p-values. However, non-significant p-values are preferred since you want your groups not to differ at baseline and therefore, using the Bonferroni correction is actually advantageous for you. So, if you follow tradition, you do not need to adjust but if you do, it will not cause any harm to your study.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thank you! Well, actually the differences shouldn't be large in randomized trials. Randomization should "equalize" subjects, make them "homogeneous". But there are also a lot of observational studies, where no constraints are forced, no randomization exists due to the fact, that the data come from medical records. Then, if a strong heterogeneity exists, then it's a good topic for a discussion of analysis done is sub-groups, on using the variables as covariates in models, or launch a dedicated study on a selected group. My case is exactly the non-randomized one. Thank you! $\endgroup$
    – Gattaca332211
    Commented Apr 10, 2022 at 13:34
  • $\begingroup$ You are right and this is why researchers usually adjust for these variables in the analyses. However, since these variables are not necessarily independent, it may be pointless to add the univariate analysis, in my opinion. Nevertheless, if the journal wants is, it causes no real harm. $\endgroup$
    – Aolon
    Commented Apr 11, 2022 at 5:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.