Adjust for covariates in small sample size (IPTW, PS match, etc.)

Question

I have a dataset of patients with a grouping variable (groups A (control) and group B (treatment)). The two groups have sample sizes of 170 vs. 30. I would like to compare outcomes between the two treatment groups but they differ in baseline covariates. I tried propensity score matching and inverse probability treatment weights (IPTW) but both don't seem to achieve good covariate balance between the groups (as of SMD <0.1).

My guess is the sample size of the treatment group is too small. Does anybody have a recommendation how I can adjust for baseline covariates in such a small sample size?

Answer 1

Differences in baseline covariates are not intrinsically a bad thing. Given the small sample size, and the possible number of "covariates" compared, it's almost a certainty one or more will show "imbalance" - but that's just an artifact of an uncontrolled familywise error rate - and hey, what's the primary endpoint anyway?

It's most useful to think of the randomization assumption not as balancing a possibly infinite list of covariates, but balancing their "average effect" on each group; exactly the rationale for propensity matching. Stephen Senn has written extensively on this issue as one of the major "myths" of clinical trials - and we must infer by the convention of "treated" and "control" that this is a randomized study. https://onlinelibrary.wiley.com/doi/abs/10.1002/sim.5713

Dr. Senn goes on the explain that imbalance is much different from confounding, since even if the variable is associated with the outcome, it is theoretically unrelated to randomization, unless there is some kind of conditional analysis such as in the "per protocol" population. In spite of all this, adjustment prevails as the superior way to handle imbalance because, given the variable conditioned upon is prognostic - i.e. it actually predicts the outcome - the tendency is to increase power. Variables that are not strongly predictive of the response tend to attenuate the power of the test when adjusted willy nilly.

But all of this goes in the bin if it's not actually a prespecified analysis. This is because you can't choose the analysis that you finally do based upon what you see in the data. Do the analysis you planned, it's probably the right one.

Answer 2

Look to see if any of these imbalanced covariates is associated with the outcome variable through regression on the available data or via a literature review. If there is little evidence to suggest they are associated, then you would feel that much more confident there is no harm in having the imbalance. You may end up choosing to balance only one or two of the covariates that indeed show evidence of an association with the endpoint. Ultimately you sill still need the "no unmeasured confounders" preface if you are going to make causal inference. See this thread.

You could perform several sensitivity analyses to see the effect of balancing each covariate separately and in combination, sample size permitting.

Also look at this thread regarding imbalance in patient characteristics due to post-baseline events.