I am an economics student with some experience with econometrics and R. I would like to know if there is ever a situation where we should include a variable in a regression in spite of it not being statistically significant?
$\begingroup$
$\endgroup$
6
-
1$\begingroup$ In medical research you would include it if it involves qualitative interactions. See the work of Lacey Gunter that I have referenced here before. Also the book by Chakraborty and Moodie published by Springer in 2013. The title is Statistical Methods for Dynamic Treatment Regimes: Reinforcement Learning, Causal Inference, and Personalized Medicine. $\endgroup$– Michael R. ChernickCommented Apr 2, 2017 at 21:48
-
14$\begingroup$ Consider also that statistical significance is completely arbitrary. What's significant? 0.05? 0.1? 0.001? If the theoretical foundation exists to include a predictor, that's reason enough to keep it. $\endgroup$– AsheCommented Apr 3, 2017 at 13:30
-
3$\begingroup$ When you say "not statistically significant" you realize that is at the 5% level of confidence, which is an arbitrary choice? (And the more variables there are, you incur the Multiple Testing Problem). $\endgroup$– smciCommented Apr 3, 2017 at 21:46
-
1$\begingroup$ @smci 0.05 = 5% level of significance corresponds to 95% level of confidence, reason enough to avoid mixing the terms in the same sentence. As there are significance procedures without a confidence interval in sight, it usually is easiest to use whichever term is more pertinent. The exceptions are when you are explaining the link at an introductory level. $\endgroup$– Nick CoxCommented Apr 4, 2017 at 6:07
-
$\begingroup$ @NickCox: thanks for the terminological correction, but my point stands that any significance-level threshold is arbitrary, like Ashe said. $\endgroup$– smciCommented Feb 7, 2022 at 2:44
|
Show 1 more comment
8 Answers
$\begingroup$
$\endgroup$
3
Yes!
That a coefficient is statistically indistinguishable from zero does not imply that the coefficient actually is zero, that the coefficient is irrelevant. That an effect does not pass some arbitrary cutoff for statistical significance does not imply one should not attempt to control for it.
Generally speaking, the problem at hand and your research design should guide what to include as regressors.
Some Quick Examples:
And do not take this as an exhaustive list. It's not hard to come up with tons more...
1. Fixed effects
A situation where this often occurs is a regression with fixed effects.
Let's say you have panel data and want to estimate $b$ in the model:
$$ y_{it} = b x_{it} + u_i + \epsilon_{it}$$
Estimating this model with ordinary least squares where $u_i$ are treated as fixed effects is equivalent to running ordinary least squares with an indicator variable for each individual $i$.
Anyway, the point is that the $u_i$ variables (i.e. the coefficients on the indicator variables) are often poorly estimated. Any individual fixed effect $u_i$ is often statistically insignificant. But you still include all the indicator variables in the regression if you are taking account of fixed effects.
(Further note that most stats packages won't even give you the standard errors for individual fixed effects when you use the built-in methods. You don't really care about significance of individual fixed effects. You probably do care about their collective significance.)
2. Functions that go together...
(a) Polynomial curve fitting (hat tip @NickCox in the comments)
If you're fitting a $k$th degree polynomial to some curve, you almost always include lower order polynomial terms.
E.g. if you were fitting a 2nd order polynomial you would run:
$$ y_i = b_0 + b_1 x_i + b_2 x_i^2 + \epsilon_i$$
Usually it would be quite bizarre to force $b_1 = 0$ and instead run $$ y_i = b_0 + b_2 x_i^2 + \epsilon_i$$
but students of Newtonian mechanics will be able to imagine exceptions.
(b) AR(p) models:
Let's say you were estimating an AR(p) model you would also include the lower order terms. For example for an AR(2) you would run:
$$ y_t = b_0 + b_1 y_{t-1} + b_2 y_{t-2} + \epsilon_t$$
And it would be bizarre to run: $$ y_t = b_0 + b_2 y_{t-2} + \epsilon_t$$
(c) Trigonometric functions
As @NickCox mentions, $\cos$ and $\sin$ terms similarly tend to go together. For more on that, see e.g. this paper.
More broadly...
You want to include right-hand side variables when there are good theoretical reasons to do so.
And as other answers here and across StackExchange discuss, step-wise variable selection can create numerous statistical problems.
It's also important to distinguish between:
- a coefficient statistically indistinguishable from zero with a small standard error.
- a coefficient statistically indistinguishable from zero with a large standard error.
In the latter case, it's problematic to argue the coefficient doesn't matter. It may simply be poorly measured.
-
$\begingroup$ Touching upon your first example, the reason we keep $u_i$ in the model seems to be that the interpretation of $b$ changes whether $u_i$ is in the model or not. (See e.g. en.wikipedia.org/wiki/Partial_regression_plot --we use something like the phrase "controlling for the linear effects of $u_i$"). In this situation, we don't have $u_i$ in the model for it's significance, we have it for the interpretation it gives us. $\endgroup$ Commented Apr 3, 2017 at 16:50
-
6$\begingroup$ Some very good answers which nevertheless already overlap a bit too much, so I'll confine my examples to a comment here. Polynomial fitting: most commonly, a quadratic should almost always be fitted by a double act of linear and squared terms. Even if only one term is significant at conventional levels, their joint effect is key. Trigonometric predictors Similarly, sine and cosine usually belong together even if one fails to qualify at conventional levels. Double acts should be fitted as such. $\endgroup$– Nick CoxCommented Apr 3, 2017 at 21:02
-
$\begingroup$ In the case of AR(p) I think it depends a lot on the context. AR(p) are often used as reduced-form models (not interpreted in a structural sense) or as benchmarks in forecasting where we do not hesitate to drop variables that are contributing a lot to variance while doing little to reduce bias. $\endgroup$ Commented Feb 5, 2022 at 18:32
$\begingroup$
$\endgroup$
3
Yes, there are. Any variable that could correlate with your response variable in a meaningful way, even at a statistically insignificant level, could confound your regression if it is not included. This is known as underspecification, and leads to parameter estimates that are not as accurate as they could otherwise be.
https://onlinecourses.science.psu.edu/stat501/node/328
From the above:
A regression model is underspecified (outcome 2) if the regression equation is missing one or more important predictor variables. This situation is perhaps the worst-case scenario, because an underspecified model yields biased regression coefficients and biased predictions of the response. That is, in using the model, we would consistently underestimate or overestimate the population slopes and the population means. To make already bad matters even worse, the mean square error MSE tends to overestimate σ², thereby yielding wider confidence intervals than it should.
-
5$\begingroup$ That is not quite true. In order to be a confounding variable it needs to cause the explained variable and the explanatory variable(s) of interest. If the explanatory variables of interest cause the variable, and it influences the outcome, then it is an intervening variable, and you should not control for it (unless you want to decompose the total effect). $\endgroup$ Commented Apr 3, 2017 at 9:58
-
1$\begingroup$ This is a very inadequate discussion on the topic of control of confounding. Correlation with the outcome is not a sufficient condition for confounding and can lead to misspecification of causal models by controlling for mediators: This leads to fallacies such as "smoking cessation does not reduce cardiovascular disease risk after controlling for coronary arterial calcium (CAC)". CAC is the primary way smoking gives you heart disease. See Causality by Pearl, 2nd ed, chapter 3 section 3. $\endgroup$– AdamOCommented Apr 3, 2017 at 15:14
-
$\begingroup$ Feel free to edit. I didn't think he was looking for that kind of depth in the answer, apologies if my brevity led to gross inaccuracy. $\endgroup$ Commented Apr 3, 2017 at 16:43
$\begingroup$
$\endgroup$
1
Usually you do not include or exclude variables for linear regression because of their significance. You include them because you assume that the selected variables are (good) predictors of the regression criteria. In other words, the predictor selection is based on theory.
Statistical insignificance in linear regression can mean two things (of which I know):
- The insignificant predictors are not related to the criteria. Exclude them but keep in mind that the insignificance does not prove that they are unrelated. Check your theory.
- The predictors are insignificant because they can be expressed as a function of other predictors. The set of predictors is then called multicollinear. This does not make the predictors "bad" in any sense but redundant.
A valid reason to exclude insignificant predictors is that you are looking for the smallest subset of predictors that explain the criteria variance or most of it. If you have found it check your theory.
-
$\begingroup$ [P]redictors of the regression criteria? You might wish to rephrase this. $\endgroup$ Commented Apr 3, 2017 at 14:10
$\begingroup$
$\endgroup$
3
In econometrics this happens left and right. For instance, if you are using quarterly seasonality dummies Q2,Q3, and Q4, it happens often that as a group they're significant, but some of them are not significant individually. In this case you usually keep them all.
Another typical case is interactions. Consider a model $y\sim x*z$, where main effect $z$ is not significant but the interaction $x*z$ is. In this case it's customary to keep the main effect. There are many reasons why you should not drop it, and some of them were discussed in the forum.
UPDATE: Another common example is forecasting. Econometrics is usually taught from inference perspective in economics departments. In inference perspective a lot of attention is on p-values and significance, because you're trying to understand what causes what and so on. In forecasting, there's not much emphasis on this stuff, because all you care is how well the model can forecast the variable of interest.
This is similar to machine learning applications, btw, which are making their way into economics recently. You can have a model with all significant variables that doesn't forecast well. In ML it's often associated with so called "over fitting". There's very little use of such model in forecasting, obviously.
-
1$\begingroup$ This seems a little exaggerated at some points. For example, it's evident even to me as a non-economist from textbooks alone that forecasting has been widely taught to economists for at least some decades. Whether there has been a "recent" (meaning precisely?) increase is a more subtle point which I leave to insiders. $\endgroup$– Nick CoxCommented Apr 3, 2017 at 20:53
-
$\begingroup$ @NickCox, agreed, it sounded as if there was no forecasting at all in curricula, which is not true. $\endgroup$– AksakalCommented Apr 3, 2017 at 20:58
-
$\begingroup$ The forecasting context is exactly where we do not hesitate to exclude variables regardless of their significance. We would especially tend to exclude ones with p-values above ~0.16 (corresponding to the AIC cutoff, with the BIC cutoff being below that and thus covered by that, too). $\endgroup$ Commented Feb 5, 2022 at 18:35
$\begingroup$
$\endgroup$
You are asking two different questions:
- When does statistical significance not matter?
- When should we include a variable in a regression in spite of it not being statistically significant?
Edit: this was true about the original post, but might no longer be true after the edits.
Regarding Q1, I think it is on the border of being too broad. There are many possible answers, some already provided. One more example is when building models for forecasting (see the source cited below for an explanation).
Regarding Q2, statistical significance is not a sound criterion for model building. Rob J. Hyndman writes the following in his blog post "Statistical tests for variable selection":
Statistical significance is not usually a good basis for determining whether a variable should be included in a model, despite the fact that many people who should know better use them for exactly this purpose. <...> Statistical tests were designed to test hypotheses, not select variables.
Also note that you can often find some variables that are statistically significant purely by chance (the chance being controlled by your choice of the significance level). The observation that a variable is statistically significant is not enough to conclude that the variable belongs in the model.
$\begingroup$
$\endgroup$
1
I'll add another "yes". I've always been taught -- and I've tried to pass it along -- that the primary consideration in covariate choice is domain knowledge, not statistics. In biostatistics, for instance, if I'm modelling some health outcome on individuals, then no matter what the regression says, you'll need some darn good arguments for me not to include age, race, and sex in the model.
It also depends on the purpose of your model. If the purpose is gaining better understanding of what factors are most associated with your outcome, then building a parsimonious model has some virtues. If you care about prediction, and not so much about understanding, then eliminating covariates may be a smaller concern.
(Finally, if you're planning to use statistics for variable selection, check out what Frank Harrell has to say on the subject -- http://www.stata.com/support/faqs/statistics/stepwise-regression-problems/, and his book Regression Modeling Strategies. Briefly, by the time you've used stepwise or similar statistically-based strategies for choosing the best predictors, then any tests of "are these good predictors?" are terribly biased -- of course they're good predictors, you've chosen them on that basis, and so the p values for those predictors are falsely low.)
-
1$\begingroup$ @EconJohn And model F statistics, $R^{2}$s, and effect estimates falsely high... and stepwise-selected models about equally likely to retain true predictors and false predictors, and about equally likely to remove true predictors and false predictors. $\endgroup$– AlexisCommented Apr 3, 2017 at 21:00
$\begingroup$
$\endgroup$
0
The only thing that the result of "statistical insignificance" truly says is that, at the selected level of Type I error, we cannot even tell whether the effect of the regressor on the dependent variable is positive or negative (see this post).
So, if we keep this regressor, any discussion about its own effect on the dependent variable does not have statistical evidence to back it up.
But this estimation failure does not say that the regressor does not belong to the structural relation, it only says that with the specific data set we were unable to determine with some certainty the sign of its coefficient.
So in principle, if there are theoretical arguments that support its presence, the regressor should be kept.
Other answers here provided specific models/situations for which such regressors are kept in the specification, for example the answer mentioning the fixed-effects panel data model.
$\begingroup$
$\endgroup$
You may include a variable of particular interest if it is the focus of research, even if not statistically significant. Also, in biostatistics, clinical significance is often different than statistical significance.