Adding a covariate to a meta-regression with low sample size

Question

In an meta-regression I am working on an interesting question has come up.

I am are running a meta-regression with 32 study sites. I have four theoretically motivated covariates I want to examine. All four have minor effect sizes with p < 0.05.

However, there is some concern about confounding. Thus, I adjusted the analyses for a binary covariate (where only 2 studies had value 1, the rest having value 0 - so highly skewed). When adjusting for this covariate, effect sizes changed by 17-40% and three covariates no longer have p < 0.05. Effectively, I am running a multiple meta-regression with the covariate of interest adjusted for potential confounding.

I have discussed this matter with colleagues. While one argument is that the adjusted models report improved estimates, another is that the regression models report more uncertain estimates due to inclusion of an extra covariate, hence the covariates have higher p-values due to more uncertainty in the model not removal of confounding. Thus, there seem to be sound conflicting theoretical and statistical arguments.

I am interested in a better understanding of the statistical aspects of this. I have tried to look up statistical literature on this, but I have not found anything that is spot on. Hopefully some of you could weigh in or provide some references.

Answer 1

My guess is that the confounder is (strongly) correlated to your original covariates, i.e., the two studies that have 1 are systematically different from the other studies also regarding the other covariates (or most of them). In that case it can happen that what was seemingly explained before by the other covariates is now largely explained by the confounder, leading to insignificant p-values.

The problem here is that if covariates are strongly correlated, this means that they share to some extent the same information, and the data cannot distinguish which variable is actually responsible (in a causal sense) for this contribution. This means that standard errors of estimators and p-values may become larger, although if indeed the confounder adds relevant information (you didn't write what the confounder's p-value is), the model will become better in terms of data fit and predictive strength. There are two different aims here, having a model that fits well and making statements about variable importance. The model with additional confounder will in all likelihood be the better model regarding fit, however because the confounder is correlated with the other covariates, it will have difficulties to attribute effects precisely to the covariates, leading to higher standard errors of some or all parameter estimates.

However, this shouldn't make you believe that the model without the confounder is better, because that only hides the problem rather than solving it. If you don't have the confounder in the model, you can't tell either to what extent the information in your original covariates may really have been caused by the confounder.