5
$\begingroup$

I have been reading medical journals and they repeatedly show baseline characteristics of samples from a randomised controlled trial, which they have then tested to ensure no differences between the two groups under study. For example in the operation group you have 39 males and 3 females. And in the non-op group you have 35 males and 8 females. The males would have a p-value, and then the females would have a p-value.

I was wandering what exactly is being tested and how. My initial thoughts were a chi-squared test, although this would produce only 1 p-value, and this would be looking at the distribution of patients between the two groups. The papers seem to suggest they assess just the females distribution, and then separately the males distribution. But is this right and how would one do it?

$\endgroup$
7
$\begingroup$

It is generally thought nowadays that testing for baseline differences in randomized experiments is misleading. Stephen Senn's book Statistical Issues in Drug Development discusses this. One of the many issues involved is that you never know when to stop. How many uncollected variables do you go back and collect in order to test for balance? Couple that with low power and no real interpretation and it's a waste of time unless cheating is suspected.

$\endgroup$
5
$\begingroup$

This is not actually an answer to your question but the logic behind such tests seems fundamentally misguided, no matter what the specifics are.

If treatment assignment is not or cannot be properly randomized, showing that both groups have approximately the same characteristics on some arbitrary set of variables is not going to replace randomization. If treatment assignment is in fact properly randomized, tests on demographic characteristics provide absolutely no information. At the conventional level, one in twenty tests should be significant because the null hypothesis is true by construction.

Furthermore, why should you care about the groups' composition? If variables like age or gender do not interact with the treatment, it does not matter in the least. If, on the other hand, you have reasons to believe your treatment does not produce the same effect in different subgroups, you will loose power but randomization ensures that it does not threaten inferences. At the same time, even groups with exactly the same composition will not help you improve power or understand the effect in each subgroup. For that, you need to include the relevant variable in the model or estimate the effect separately on each subgroup.

In any case, interpreting a large p-value as evidence that there is no difference is mistaken, especially for such small sample sizes. If you consider a situation in which treatment cannot be randomized, the power of a test for differences in age or gender will of course depend a lot on sample size. With a small sample, you have basically no power to detect anything but obvious differences, even if smaller differences do matter. With a large sample, you will find small differences (e.g. a few months differences in age) to be “significant” even if they are so small so as to have absolutely no effect on your outcome.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.