I have multiple ($12$) predictors ($X$) for an outcome (spending) where it's likely/possible that:
- Some predictors are correlated
- Some predictors could (partially) mediate the effect of others
- There may be unobserved confounding
The theory in this field is weak/poorly developed and doesn't help that much in determining the best model.
Empirically, we see quite marked parameter shifts when including/excluding certain variables in different models, which suggests to me that there are suppressor/confounding/other weird effects, etc going on.
If we assume that the model $y \sim X$ is misspecified, at least to some extent, is it reasonable to use all-possible-subsets regression to hint at which parameters are suspect? For example, if we fit the set of all models entering $1$ to $k$ of the predictors in $X$, we could plot how much shifting there is of parameter values when other variables enter or exit the model. We'd keep $k$ less than the total number of predictors to make it tractable.
My intuition is that predictors for these coefficients shift a lot when adding/removing other variables are suspect in some way! If not that they are necessarily misspecified themselves, then certainly that this would indicate areas to be cautious/explore.
Is this a reasonable thing to do? The attached plot, for example, shows that some variables in this model do shift quite a lot more than others as items from $X$ are added/removed: the point is the mean of coefficients from all models in which the predictor was entered; the error bar is the 95% hdci for the parameter across all models. I should also note these are zero-inflated binomial models in which predictors are also used to predict zero outcomes so we have two parameters for each predictor.
Is there any literature on this kind of thing? Searching, I find things on omitted variable bias and DAG selection — but this is kind of the opposite... I know this model is probably wrong but would like to use the parameters to tell me where to look first to fix it.
EDIT 1/Jun - following the helpful comments below
@FrankHarrell using a DAG is great advice and is definitely where we want to go. I can sense a frustration that we're trying to cut corners/avoid thinking about the hard problem, but I wonder if you have misinterpreted the intention here?
The idea here is actually to problematise the existing literature which isn't done this way, and to highlight (some) areas of particular concern. If a parameter did shift markedly within this subset of models is that not evidence that inferences based on bivariate relationships (or models including subsets of variables) are unsafe?
This is the first study to have measured a full-ish set of variables which have previously been measured individually. I'd hoped that by dredging all possible we'd be able to show which current findings were least-safe — in the sense that they definitely depend on how you define the model. This would be a preliminary (motivational) step to drive us towards building a DAG.
Having played around with a little simulation, this method does seem able to detect very simple examples of mis-specification. For example, if you simulate data in lavaan using something like:
y ~ .3*a + .1*b + .1*c + 0*z z ~ .3*a
Then applying this technique produces a plot like this:
That seems to have some heuristic value as a guide to future research, even if you can't draw the DAG from it. And it provides a visual way to explain to less quantitatively-focussed researchers that there is a problem with current estimates of
z. I can see there is a risk of overfitting, but would you still be bearish about presenting this even with a the much more constrained objectives described?