So I run the multivariate model and the significance levels decrease (i.e., larger p values) for almost all predictors. The extent to which they decrease vary, some may go from <0.001 to insignificant.

There have been discussions of CV of this issue (e.g., Significant predictors become non-significant in multiple logistic regression).

But I thinks one missing part across these discussion is: What can we comment about the model specification in that multivariate model, such as the selection of that specific set of predictors? Is it a good specification (because we identify relevant confounders so that, when included, they appropriately adjust the significance levels and give us a better understanding of the true relationship, which is none in this example)? Can we stop there after some effect of interest becomes insignificant and make our conclusion (no effect) or continue with model specification or, more generally, analysis in some way?

Any thought is appreciated.

  • $\begingroup$ When you say 'significance levels', do you ihstead mean p-values? $\endgroup$ – Glen_b Apr 6 '15 at 2:06
  • $\begingroup$ @Glen_b Yes, that's what I meant. $\endgroup$ – NonSleeper Apr 6 '15 at 3:13

First, selection of covariates into the model should be based on some kind of theory or previous empirical work, rather than on purely mathematical relationships within the data (we should establish hypothesized relationships a priori, rather than going on a "fishing expedition" for significance). Understanding the hypothesized relationship between the covariate and the outcome in a bivariate relationship as well as the relationship between your covariates can help you understand why you ended up with a model that had zero statistically significant coefficients even though all of those variables were significant in bivariate relationships.

Rather than assume that my lack of significance was due to "good" specification, if I saw that all of my estimates all became not statistically significant in the full model--especially my main predictor of interest--I would be concerned about model misspecification. I would check for a couple things, such as (not in any particular order):

  1. Multicollinearity: If two or more of your covariates are closely related to one another, you don't need all of them in there

  2. Endogeneity: This is a big problem that can cause inconsistent estimates. Are there confounders you have not accounted for (unobserved confounders/omitted variable bias)? Are there covariates that were not properly measured?

  3. Number of observations: Do any of your covariates have missing values? How many observations dropped out of the analysis in the estimation of the full model? If a large proportion of your original sample was excluded from analysis, the subsample used in estimation may be biased. That is, the coefficients were not estimated on the full sample, rather just the subsample that has non-missing values for all covariates, and this subsample may be systematically different from the excluded subsample. If the true relationship is positive, but the relationship within the subsample is negative, you may get null results.

Only after I have checked the above possibilities (and consult with others about things I may have missed) would I be "confident" that my model was correctly specified and indeed the true relationship is null. However, as I stated above, if the theory predicts that Y will increase with X and I find null results after controlling for W, S, and Z, I would not automatically make the assumption that the specification is "good" (I would assume the opposite and try to find out the reason for my null results).

I talked at length above about using theory, etc. but in my experience, different fields handle whether to include non-significant variables in the model: my former biostatistics professors favored parsimonious models, whereas my former econometrics professors favored models that controlled for theoretically important variables (whether significant or not). My former epidemiology professors were somewhere in between: they had strong reservations about keeping variables based solely on p-values, but they didn't control for as many things as the econometricians.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.