3
$\begingroup$

I am running various models in R for sake of prediction. If I run a model and a specific variable is showing itself to be insignificant (say, at the alpha=0.05 level), would I want to simply discard this variable? Would my next step be to remove the variable from the model and re-run that same type of model, then see how drastic the affect was on the R-squared and predictive accuracy?
Then, run an ANOVA comparison to see if it is true that the smaller, nested model is just as "good" as the larger model? Thoughts? Is it always safe to simply remove insufficiently-significant variables?

$\endgroup$
  • 1
    $\begingroup$ The question you have to ask yourself is: why is that variable in there in the first place? $\endgroup$ – Wayne Mar 12 '16 at 5:16
  • $\begingroup$ When you say your goal is prediction, what do you mean? Do you actually care about taking in some new inputs $x^*$ and spitting out the most "accurate" output $y^*$? If so, how would you define accurate? If you instead want to do inference on the parameters of the model, then that is a different task.. $\endgroup$ – user44764 Mar 12 '16 at 5:18
  • 1
    $\begingroup$ In general doing model selection on the basis of $p$-values can be a dubious practice. You say your goal is prediction, so why not use prediction accuracy directly as your criterion for including variables in a model? $\endgroup$ – dsaxton Mar 12 '16 at 6:03
  • 1
    $\begingroup$ Well i have stumbled upon cases where two variables werent statistically significant by themselves, however their interaction led to higher predictive performance. It depends on what kind of models you are using. $\endgroup$ – amanita kiki Mar 12 '16 at 8:16
7
$\begingroup$

I am also not quite understand what "running various models in R for sake of prediction" means, I am just assuming that you want to do a hypothesis testing and conduct model selection. For example, you are doing a linear regression in R, you regress one response variable on several independent variables. I am assuming that your response variables are continuous. The summary output in R will give the p-value for each of the variables, some of them are significant at $\alpha=0.05$, and some of them might not. Then you are not sure if you can just delete the variables that have insignificant p-value and refit this model. You plug a set of new independent variables into this model, the returned $y$ is your predictive response variable.

Your question would I want to simply discard this variable?

Simple word: No, you never throw away any variables that are not significant. Even if the significance level of all the independent variables shows that the variables are insignificant, it does not mean that any of those independent variables won't affect the response variable at all.

The ANOVA test or the summary output in R are all simultaneous testing, this means that you are hypothetical over a set of coefficients instead of a single one. For instance, given a linear model $y=\alpha + \beta_1 x_1 + \beta_2 x_2 + \ldots + \beta_p x_p + \epsilon$, the $H_0: \beta_1 = \beta_2 =\ldots=\beta_p=0$, and $H_A: \beta_1 \neq \beta_2 \neq\ldots \neq \beta_p \neq 0$. In the ANOVA, the significance level of F test will determine if we want to reject the $H_0$ or not. This is a simultaneous testing. If F-statistic is significant, this means that all the $\beta_p$ are not equal to 0; Otherwise, at least one $\beta_p$ is not equal to 0.

After understanding the simultaneous test, let's look at the summary output, summary(name of a linear model object), I think this is the place that you want to discard a variable in terms of the p-value. Still use the example above, $y=\alpha + \beta_1 x_1 + \beta_2 x_2 + \ldots + \beta_p x_p + \epsilon$, the summary() output will have p-value corresponding to all the variables, those p-values are calculated from the t-distribution. After discarding one variable, the p-value in this new model will have different meaning compared to the model above. The hypothesis test for the F-test is different. There is no correction for this model. When you repeat this procedure for multiple times, and compare your final selected model to your original model, it compares orange to apple. However, there is one possible scenario that you can throw one variable given all the other variables, it is when their design matrix is orthogonal. But this situation is very rare. In summary, discarding the insignificant variables and stepwise model selections (backward, forward and bidirectional) all have this kinda criticism.

Here are several methods to fix this:

  1. Sequential F test
  2. Multi-comparison method: fix the significance level and control the type I error before doing the test. There are many methods on this: Tukey, Dunnett, Bonferroni, Scheffé method for ALL linear comparisons, Hsu.
  3. Bayesian Hypothesis testing
$\endgroup$
  • $\begingroup$ +1, with the clarification that "you never throw away any variables that are not significant" refers to variables that have theoretical, principled reasons for being in there in the first place. Hence my original comment to the OP. If it makes no sense for the variable to have been included in the first place, hopefully we're lucky enough that it ends up with a low p-value. $\endgroup$ – Wayne Mar 12 '16 at 16:45
  • $\begingroup$ Thank you, all! @Wayne, you bring up a very nice point- yes, I do think that my variables all have a logical reason to be there and I don't want to do away with them. I worked with reducing models before, but I haven't ever had a model where I kept all of the variables; I wasn't sure if it was illogical to keep variables that showed low significance. Shijia Bian, thank you also for your response! I will look into all of your links! $\endgroup$ – Nameless Mar 14 '16 at 16:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.