Do I throw away a variable that is not statistically significant? I am running various models in R for sake of prediction.  If I run a model and a specific variable is showing itself to be insignificant (say, at the alpha=0.05 level), would I want to simply discard this variable? 
Would my next step be to remove the variable from the model and re-run that same type of model, then see how drastic the affect was on the R-squared and predictive accuracy?
Then, run an ANOVA comparison to see if it is true that the smaller, nested model is just as "good" as the larger model?
Thoughts?
Is it always safe to simply remove insufficiently-significant variables?
 A: I am also not quite understand what "running various models in R for sake of prediction" means, I am just assuming that you want to do a hypothesis testing and conduct model selection. For example, you are doing a linear regression in R, you regress one response variable on several independent variables. I am assuming that your response variables are continuous. The summary output in R will give the p-value for each of the variables, some of them are significant at $\alpha=0.05$, and some of them might not. Then you are not sure if you can just delete the variables that have insignificant p-value and refit this model. You plug a set of new independent variables into this model, the returned $y$ is your predictive response variable.  
Your question would I want to simply discard this variable? 
Simple word: No, you never throw away any variables that are not significant. Even if the significance level of all the independent variables shows that the variables are insignificant, it does not mean that any of those independent variables won't affect the response variable at all. 
The ANOVA test or the summary output in R are all simultaneous testing, this means that you are hypothetical over a set of coefficients instead of a single one. For instance, given a linear model $y=\alpha + \beta_1 x_1 + \beta_2 x_2 + \ldots + \beta_p x_p + \epsilon$, the $H_0: \beta_1 = \beta_2 =\ldots=\beta_p=0$, and $H_A: \beta_1 \neq \beta_2 \neq\ldots \neq \beta_p \neq 0$. In the ANOVA, the significance level of F test will determine if we want to reject the $H_0$ or not. This is a simultaneous testing. If F-statistic is significant, this means that all the $\beta_p$ are not equal to 0; Otherwise, at least one  $\beta_p$ is not equal to 0. 
After understanding the simultaneous test, let's look at the summary output, summary(name of a linear model object), I think this is the place that you want to discard a variable in terms of the p-value. Still use the example above, $y=\alpha + \beta_1 x_1 + \beta_2 x_2 + \ldots + \beta_p x_p + \epsilon$, the summary() output will have p-value corresponding to all the variables, those p-values are calculated from the t-distribution. After discarding one variable, the p-value in this new model will have different meaning compared to the model above. The hypothesis test for the F-test is different. There is no correction for this model. When you repeat this procedure for multiple times, and compare your final selected model to your original model, it compares orange to apple. However, there is one possible scenario that you can throw one variable given all the other variables, it is when their design matrix is orthogonal. But this situation is very rare. In summary, discarding the insignificant variables and stepwise model selections (backward, forward and bidirectional) all have this kinda criticism.
Here are several methods to fix this:


*

*Sequential F test

*Multi-comparison method: fix the significance level and control the type I error before doing the test. There are many methods on this: Tukey, Dunnett, Bonferroni, Scheffé method for ALL linear comparisons, Hsu.

*Bayesian Hypothesis testing
