39
$\begingroup$

I have several covariates in my calculation for a model, and not all of them are statistically significant. Should I remove those that are not?

This question discusses the phenomenon, but does not answer my question: How to interpret non-significant effect of a covariate in ANCOVA?

There is nothing in the answer to that question that suggests that non-significant covariates be taken out, though, so right now I am inclined to believe that they should stay in. Before even reading that answer, I was thinking the same since a covariate can still explain some of the variance (and thus help the model) without necessarily explaining an amount beyond some threshold (the significance threshold, which I see as not applicable to covariates).

There is another question somewhere on CV for which the answer seems to imply that covariates should be kept in regardless of significance, but it is not clear on that. (I want to link to that question, but I was not able to track it down again just now.)

So... Should covariates that do not show as statistically significant be kept in the calculation for the model? (I have edited this question to clarify that covariates are never in the model output by the calculation anyway.)

To add complication, what if the covariates are statistically significant for some subsets of the data (subsets which have to be processed separately). I would default to keeping such a covariate, otherwise either different models would have to be used or you would have a statistically significant covariate missing in one of the cases. If you also have an answer for this split case, though, please mention it.

$\endgroup$
  • 6
    $\begingroup$ Generally speaking, I would say that you should keep variables that are theoretically important or that have been significant in prior studies, even though your data do not support their effect. That being said, to get a more specific answer, I think you should add a couple of lines to explain your model and its purpose (e.g. identifying risk factors, making prediction, ...). $\endgroup$ – ocram Aug 3 '13 at 18:17
  • $\begingroup$ I would say it depends. Tests are just indicators. If you believe that there should be a small dependence then think about keeping in the model. If you believe as well that the dependence should not be there then leave it out. $\endgroup$ – Bene Aug 3 '13 at 18:18
  • $\begingroup$ OK, so you are both saying that non-significance does not dictate a covariate being removed from consideration, so you have both actually answered my question. I should actually rephrase my question to more clearly indicate that what I am asking is whether stastistical significance of a covariate is a necessary condition for keeping it ("Does non-significance of a covariate mean it should be removed..."), and I would accept either of your comments as answers. $\endgroup$ – A.M. Aug 3 '13 at 18:28
  • $\begingroup$ Before I do that, though, I would like to make sure I am using the right terminology. Originally I wrote "kept in the model", but that did not seem right because covariates never appear in the model. I settled for "kept in the calculation for the model" (and "removed from consideration"), but is there a better way of saying this? What is the right term for what the covariate is being kept in or removed from? $\endgroup$ – A.M. Aug 3 '13 at 18:30
  • 3
    $\begingroup$ You would need to validate the correct performance of such selection procedures. Others have failed. $\endgroup$ – Frank Harrell Aug 3 '13 at 22:59
32
$\begingroup$

You have gotten several good answers already. There are reasons to keep covariates and reasons to drop covariates. Statistical significance should not be a key factor, in the vast majority of cases.

  1. Covariates may be of such substantive importance that they have to be there.
  2. The effect size of a covariate may be high, even if it is not significant.
  3. The covariate may affect other aspects of the model.
  4. The covariate may be a part of how your hypothesis was worded.

If you are in a very exploratory mode and the covariate is not important in the literature and the effect size is small and the covariate has little effect on your model and the covariate was not in your hypothesis, then you could probably delete it just for simplicity.

$\endgroup$
  • 6
    $\begingroup$ A very important but often neglected situation is covered by #4 here, but I will spell it out. Often -- indeed usually -- you should want to compare your results with those of previous workers with similar data. If others found particular covariates worth including in their models, you should want to compare your results with theirs, regardless of whether your covariates achieve (conventional) significance levels. Note that cases here can vary from reporting model(s) you decide are not (especially) good to reporting model(s) you decide are good. $\endgroup$ – Nick Cox Aug 4 '13 at 9:24
  • 1
    $\begingroup$ I was definitely leaning toward 'keep in' (and not making much of p-value for covariates in the first place), but your answer makes a very nice checklist (well...two) for a minority to take out. The effect size is something I had not considered, and while I did consider hypotheses I very much like that you included it, for the reasons @NickCox mentioned and simply to discouraging fishing. $\endgroup$ – A.M. Aug 4 '13 at 14:21
25
$\begingroup$

The long answer is "yes". There are few reasons to remove insignificant predictors and many reasons not to. As far as interpreting them you do so ignoring the $P$-value just as you might interpret other predictors: with confidence intervals for effects over interesting ranges of the predictor.

$\endgroup$
  • 10
    $\begingroup$ The long answer is "yes" ! +1 and a LOL. $\endgroup$ – Peter Flom Aug 3 '13 at 19:44
  • $\begingroup$ If not p-values, what are other reasons to remove predictors? You mention interpreting confidence intervals, but it seems like an "interesting range" would be zero, which means people would interpret CIs much like p-values (inclusion or exclusion of zero). $\endgroup$ – Mark White Jun 23 '17 at 16:54
  • 1
    $\begingroup$ What are reasons to remove predictors when this distorts statistical properties? Not clear on your question and the "zero". $\endgroup$ – Frank Harrell Jun 23 '17 at 18:56
7
$\begingroup$

One useful insight is that there is really nothing specific about a covariate statistically speaking, see e.g. Help writing covariates into regression formula. Incidentally, it might explain why there is no covariate tag. Consequently, material here and elsewhere about non-significant terms in a linear model are relevant, as are the well known critics of stepwise regression, even if ANCOVA is not explicitly mentioned.

Generally speaking, it's a bad idea to select predictors based on significance alone. If for some reason you cannot specify the model in advance, you should consider other approaches but if you planned to include them in the first place, collected data accordingly and are not facing specific problems (e.g. collinearity), just keep them.

Regarding the reasons to keep them, the objections you came up with seem sound to me. Another reason would be that removing non-significant predictors biases inferences based on the model. Yet another way to look at all this is to ask what would be gained by removing these covariates after the fact.

$\endgroup$
4
$\begingroup$

We really need more information about your goals to answer this question. Regressions are used for two main purposes:

  1. Prediction
  2. Inference

Prediction is when your goal is to be able to guess at values of the outcome variable for observations that are not in the sample (although usually they are within the range of the sample data–otherwise, we sometimes use the word "forecasting"). Prediction is useful for advertising purposes, finance, etc. If you are just interested in predicting some outcome variable, I have little to offer you.

Inference is where the fun is (even if it is not where the money is). Inference is where you are trying to make conclusions about specific model parameters–usually to determine a causal effect of one variable on another. Despite common perception, regression analysis is never sufficient for causal inference. You must always know more about the data generating process to know whether your regression captures the causal effect. The key issue for causal inference from regressions is whether the conditional mean of the error (conditional on the regressors) is zero. This cannot be known from p-values on regressors. It is possible to have regression estimators that are unbiased or consistent, but that requires far more effort than just throwing some obvious controls into the regression and hoping you got the important ones. The best coverage I have seen of approaching causal inference with observational data is in two books by Angrist and Pischke (Mastering 'Metrics: The Path from Cause to Effect and Mostly Harmless Econometrics). Mastering Metrics is the easier read and is quite cheap, but be warned that it is not a treatment of how to do regressions but rather of what they mean. For a good coverage of examples of good and bad observational research designs, I recommend David Freedman's (1991) "Statistical Models and Shoe Leather", Sociological Methodology, volume 21 (a short and easy read with fascinating examples).

Aside: the obsession with statistical technique over good research design in most college courses is a pedagogical peeve of mine.

Second aside to motivate the current importance of this issue: the difference between prediction and inference is why big data are not a substitute for science.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.