I have multiple ($12$) predictors ($X$) for an outcome (spending) where it's likely/possible that:

  • Some predictors are correlated
  • Some predictors could (partially) mediate the effect of others
  • There may be unobserved confounding

The theory in this field is weak/poorly developed and doesn't help that much in determining the best model.

Empirically, we see quite marked parameter shifts when including/excluding certain variables in different models, which suggests to me that there are suppressor/confounding/other weird effects, etc going on.

If we assume that the model $y \sim X$ is misspecified, at least to some extent, is it reasonable to use all-possible-subsets regression to hint at which parameters are suspect? For example, if we fit the set of all models entering $1$ to $k$ of the predictors in $X$, we could plot how much shifting there is of parameter values when other variables enter or exit the model. We'd keep $k$ less than the total number of predictors to make it tractable.

My intuition is that predictors for these coefficients shift a lot when adding/removing other variables are suspect in some way! If not that they are necessarily misspecified themselves, then certainly that this would indicate areas to be cautious/explore.

Is this a reasonable thing to do? The attached plot, for example, shows that some variables in this model do shift quite a lot more than others as items from $X$ are added/removed: the point is the mean of coefficients from all models in which the predictor was entered; the error bar is the 95% hdci for the parameter across all models. I should also note these are zero-inflated binomial models in which predictors are also used to predict zero outcomes so we have two parameters for each predictor.

Is there any literature on this kind of thing? Searching, I find things on omitted variable bias and DAG selection — but this is kind of the opposite... I know this model is probably wrong but would like to use the parameters to tell me where to look first to fix it.

enter image description here

EDIT 1/Jun - following the helpful comments below

@FrankHarrell using a DAG is great advice and is definitely where we want to go. I can sense a frustration that we're trying to cut corners/avoid thinking about the hard problem, but I wonder if you have misinterpreted the intention here?

The idea here is actually to problematise the existing literature which isn't done this way, and to highlight (some) areas of particular concern. If a parameter did shift markedly within this subset of models is that not evidence that inferences based on bivariate relationships (or models including subsets of variables) are unsafe?

This is the first study to have measured a full-ish set of variables which have previously been measured individually. I'd hoped that by dredging all possible we'd be able to show which current findings were least-safe — in the sense that they definitely depend on how you define the model. This would be a preliminary (motivational) step to drive us towards building a DAG.

Having played around with a little simulation, this method does seem able to detect very simple examples of mis-specification. For example, if you simulate data in lavaan using something like:

y ~ .3*a + .1*b + .1*c + 0*z
z ~ .3*a 

Then applying this technique produces a plot like this:

enter image description here

That seems to have some heuristic value as a guide to future research, even if you can't draw the DAG from it. And it provides a visual way to explain to less quantitatively-focussed researchers that there is a problem with current estimates of z. I can see there is a risk of overfitting, but would you still be bearish about presenting this even with a the much more constrained objectives described?

  • $\begingroup$ What are you interested in inference or prediction? If its inference what are your null hypotheses? $\endgroup$ – Grada Gukovic May 28 at 20:37
  • 1
    $\begingroup$ @FrankHarrell and all — many thanks for the helpful suggestions — I'm grateful for your time. I have edited the question above to make clear the constrained nature of what I'm trying to achieve here. This much more about motivating the field to build a DAG and highlight issues with existing work than making definitive inferences from these data — I should have made that clearer. $\endgroup$ – bjw Jun 1 at 12:33
  • 1
    $\begingroup$ @FrankHarrell I'm not sure if you're clear on what I want to use this for. Completely agree what we would like to know can't be learned from data alone. My question is whether we can identify things that are more complicated than currently assumed from the data? i.e. whether this has pedagogical value in highlighting areas of concern in a poorly developed/naive literature. The sim I ran suggested to me that it might, and there's nothing in the texts suggested that I can see contradicts that. Levy's book about the dangers of the phenomena I want to warn people of, and make more visible. $\endgroup$ – bjw Jun 2 at 11:03
  • 1
    $\begingroup$ I think that you can learn a little from examining relationships in observed data but since you have no notion of "the truth" it's just hard to interpret. I think you would learn more from simulation. Take a look at doi.org/10.1093/ije/dyab096 $\endgroup$ – Frank Harrell Jun 2 at 12:30
  • 2
    $\begingroup$ @FrankHarrell Many thanks for these comments and the reference. I'd like to award the points bounty (if you care about this, do duplicate your last comment in a reply and I'll be able to do that). mkt that's also a useful point... you are right that it would get much more complex with interactions. $\endgroup$ – bjw Jun 3 at 13:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.