Should covariates that are not statistically significant be 'kept in' when creating a model?

Question

I have several covariates in my calculation for a model, and not all of them are statistically significant. Should I remove those that are not?

This question discusses the phenomenon, but does not answer my question: How to interpret non-significant effect of a covariate in ANCOVA?

There is nothing in the answer to that question that suggests that non-significant covariates be taken out, though, so right now I am inclined to believe that they should stay in. Before even reading that answer, I was thinking the same since a covariate can still explain some of the variance (and thus help the model) without necessarily explaining an amount beyond some threshold (the significance threshold, which I see as not applicable to covariates).

There is another question somewhere on CV for which the answer seems to imply that covariates should be kept in regardless of significance, but it is not clear on that. (I want to link to that question, but I was not able to track it down again just now.)

So... Should covariates that do not show as statistically significant be kept in the calculation for the model? (I have edited this question to clarify that covariates are never in the model output by the calculation anyway.)

To add complication, what if the covariates are statistically significant for some subsets of the data (subsets which have to be processed separately). I would default to keeping such a covariate, otherwise either different models would have to be used or you would have a statistically significant covariate missing in one of the cases. If you also have an answer for this split case, though, please mention it.

Answer 1

You have gotten several good answers already. There are reasons to keep covariates and reasons to drop covariates. Statistical significance should not be a key factor, in the vast majority of cases.

1. Covariates may be of such substantive importance that they have to be there.
2. The effect size of a covariate may be high, even if it is not significant.
3. The covariate may affect other aspects of the model.
4. The covariate may be a part of how your hypothesis was worded.

If you are in a very exploratory mode and the covariate is not important in the literature and the effect size is small and the covariate has little effect on your model and the covariate was not in your hypothesis, then you could probably delete it just for simplicity.

Answer 2

The long answer is "yes". There are few reasons to remove insignificant predictors and many reasons not to. As far as interpreting them you do so ignoring the $P$-value just as you might interpret other predictors: with confidence intervals for effects over interesting ranges of the predictor.

Answer 3

One useful insight is that there is really nothing specific about a covariate statistically speaking, see e.g. Help writing covariates into regression formula. Incidentally, it might explain why there is no covariate tag. Consequently, material here and elsewhere about non-significant terms in a linear model are relevant, as are the well known critics of stepwise regression, even if ANCOVA is not explicitly mentioned.

Generally speaking, it's a bad idea to select predictors based on significance alone. If for some reason you cannot specify the model in advance, you should consider other approaches but if you planned to include them in the first place, collected data accordingly and are not facing specific problems (e.g. collinearity), just keep them.

Regarding the reasons to keep them, the objections you came up with seem sound to me. Another reason would be that removing non-significant predictors biases inferences based on the model. Yet another way to look at all this is to ask what would be gained by removing these covariates after the fact.

Answer 4

We really need more information about your goals to answer this question. Regressions are used for two main purposes:

1. Prediction
2. Inference

Prediction is when your goal is to be able to guess at values of the outcome variable for observations that are not in the sample (although usually they are within the range of the sample data–otherwise, we sometimes use the word "forecasting"). Prediction is useful for advertising purposes, finance, etc. If you are just interested in predicting some outcome variable, I have little to offer you.

Inference is where the fun is (even if it is not where the money is). Inference is where you are trying to make conclusions about specific model parameters–usually to determine a causal effect of one variable on another. Despite common perception, regression analysis is never sufficient for causal inference. You must always know more about the data generating process to know whether your regression captures the causal effect. The key issue for causal inference from regressions is whether the conditional mean of the error (conditional on the regressors) is zero. This cannot be known from p-values on regressors. It is possible to have regression estimators that are unbiased or consistent, but that requires far more effort than just throwing some obvious controls into the regression and hoping you got the important ones. The best coverage I have seen of approaching causal inference with observational data is in two books by Angrist and Pischke (Mastering 'Metrics: The Path from Cause to Effect and Mostly Harmless Econometrics). Mastering Metrics is the easier read and is quite cheap, but be warned that it is not a treatment of how to do regressions but rather of what they mean. For a good coverage of examples of good and bad observational research designs, I recommend David Freedman's (1991) "Statistical Models and Shoe Leather", Sociological Methodology, volume 21 (a short and easy read with fascinating examples).

Aside: the obsession with statistical technique over good research design in most college courses is a pedagogical peeve of mine.

Second aside to motivate the current importance of this issue: the difference between prediction and inference is why big data are not a substitute for science.