Please pardon me if you find this question very silly but this doubt has been troubling me for some time now whenever I want to run a regression.

I am working on SAS. I have a dataset which has 24,000 observations, and there are about 50 independent variables. There are no missing values and/or outliers. Dummy coding for categorical variables is also done. So, data preparation is complete. Now, when I run a regression model on this dataset, there are a few variables (8 variables) for which p-value is > 0.05 i.e. these variables are insignificant.

My question is what next? Do we remove these variables from the final regression equation? So, instead of having 50 independent variables, we'll have 42 independent variables (42 Beta coefficients + 1 for constant). Or do we need to remove one of these insignificant variables and re-run the regression model to see if there's any previously insignificant variable becomes significant now?


There are several ways of tackling this problem. Most people would use a variable selection technique for choosing which variables to keep and which to discard. There are a few things you have to consider before doing this:

1) Do you have a limit on the number of covariates you want to explain the change in the response? (i.e. you only want 10 variables at most)

2) Many of the variables can be strongly related, so that they don't really provide linearly independent information.

Thus, the goal of any variable selection is to removed unnecessary covariates with negligible contribution and to removed correlated covariates so that the remaining covariates provide as much independent information about the response as possible.

There a ton of details that go into variable selection, and more than can be explained in one answer. But, there are three popular types of variable selection: Forward Selection, Backward Removal, and Stepwise Regression (combines both forward and backward). Stepwise Variable Selection is generally thought of being the best (from my knowledge).

So how does one do this? It depends on your software and I do not know to do it within SAS. I can write you the basic steps for completing it, but you'll have to figure out some of it yourself in SAS.

Another thing you have to consider: Just because a variable's p-value is > 0.05, is it realistic to remove it? In some cases, we can remove variables because they are insignificant in explaining the response. But in some cases, even insignificant variables must be kept.

Probably the easiest way, but not necessarily the best, would to remove the most insignificant variable one at a time until all remaining variables are significant. Hope this helps!

  • 5
    $\begingroup$ Forward, backward & stepwise variable selection are invalid. They should not be used or recommended. For a conceptual overview, it may help to see my answer here: Algorithms for automatic model selection. $\endgroup$ Sep 1 '15 at 21:20
  • $\begingroup$ @gung While I agree with you that maybe they aren't the best methods or recommended in all cases from some of my own research, they do have some use. That or I just wasted a semester of grad school learning about them haha. $\endgroup$
    – Jake
    Sep 1 '15 at 21:28
  • 3
    $\begingroup$ To be honest, I think your having spent a semester learning about them was largely a waste. It is possible to use them validly by incorporating them into a cross-validation scheme, shrinking the coefficients & using special adjustments so that the p-values will be valid. But few people know about those methods, & they are hard to do. There are other methods that are more straight-forward to select variables without overfitting or invalidating your hypothesis tests. $\endgroup$ Sep 1 '15 at 21:32
  • $\begingroup$ Thanks Jake and gung for your valuable inputs. Since, I am not an expert on regression and I don't want R^2 of perfectly 1, Stepwise regression should also work for me. All the help is appreciated :) $\endgroup$
    – abhi1495
    Sep 3 '15 at 6:09
  • 1
    $\begingroup$ Just to clarify, make sure you aren't using R^2 as a model selection criterion. Because of the nature of R^2, it will also go up if you add more covariates, even if they don't add any new significant information. R^2 adjusted, or even AIC would be a lot better of a selection criterion. And @gung, we didn't spend the whole semester looking at stepwise regression, but it was the only method introduced as a model selection tool $\endgroup$
    – Jake
    Sep 4 '15 at 18:50

Not the answer you're looking for? Browse other questions tagged or ask your own question.