Regression - What to do with insignificant variables? Please pardon me if you find this question very silly but this doubt has been troubling me for some time now whenever I want to run a regression.
I am working on SAS. I have a dataset which has 24,000 observations, and there are about 50 independent variables. There are no missing values and/or outliers. Dummy coding for categorical variables is also done. So, data preparation is complete. Now, when I run a regression model on this dataset, there are a few variables (8 variables) for which p-value is > 0.05 i.e. these variables are insignificant.
My question is what next? Do we remove these variables from the final regression equation? So, instead of having 50 independent variables, we'll have 42 independent variables (42 Beta coefficients + 1 for constant). Or do we need to remove one of these insignificant variables and re-run the regression model to see if there's any previously insignificant variable becomes significant now?
 A: There are several ways of tackling this problem.  Most people would use a variable selection technique for choosing which variables to keep and which to discard. There are a few things you have to consider before doing this:
1) Do you have a limit on the number of covariates you want to explain the change in the response?  (i.e. you only want 10 variables at most)
2) Many of the variables can be strongly related, so that they don't really provide linearly independent information.  
Thus, the goal of any variable selection is to removed unnecessary covariates with negligible contribution and to removed correlated covariates so that the remaining covariates provide as much independent information about the response as possible.
There a ton of details that go into variable selection, and more than can be explained in one answer. But, there are three popular types of variable selection: Forward Selection, Backward Removal, and Stepwise Regression (combines both forward and backward).  Stepwise Variable Selection is generally thought of being the best (from my knowledge).  
So how does one do this?  It depends on your software and I do not know to do it within SAS.  I can write you the basic steps for completing it, but you'll have to figure out some of it yourself in SAS. 
Another thing you have to consider:  Just because a variable's p-value is > 0.05, is it realistic to remove it?  In some cases, we can remove variables because they are insignificant in explaining the response.  But in some cases, even insignificant variables must be kept.
Probably the easiest way, but not necessarily the best, would to remove the most insignificant variable one at a time until all remaining variables are significant.  Hope this helps!
