Why is this thinking incorrect? Delete insignificant covariates There are a lot of people in this site that say that you should never delete covariates from your model just because they're insignificant.
That seems quite silly to me. Just because your variable is insignificant (in the sense that the $p$-value is high), doesn't mean that it won't affect your model predictions. That insignificant variable might have a very significant estimate, which just happens to have a  big enough standard error for it to be deemed insignificant. 
But the model predictions don't care about sd.errors. They care about the estimates. So they will see that insignifcant covariate as a SIGNIFICANT covariate. 
So, unless you want insignificant stuff to cause significant bias .... delete it.
Why is that logic wrong?
For example, you might get an estimate of 5 million, with a standard error of 3 million, such that it is definitely insignificant .... but it's still going to affect your predictions!
 A: There are multiple issues at work here:


*

*Whether to simplify a model intended for prediction - or not.

*If so, whether to use p values, e.g., stepwise-regression.

*Finally, the difference between prediction and inferential statitics.


So:


*

*Recall the bias-variance tradeoff: a more complex model will have a lower bias (as you note), but also a larger variance. And a model's variance (specifically: the predictions' variance) is a component of the error. Therefore, a model can be too complex and yield a larger prediction error than a simpler model. Compare Figure 2.11 in Hastie, Tibshirani & Friedman (2008), The Elements of Statistical Learning:

Therefore it indeed makes sense to simplify a model intended for prediction.

*Given that we do want to simplify the model, are p values a good way to go about this? No, they aren't, and this has been addressed multiple times here and elsewhere. It's far better to impose sparsity via the Lasso, see Hastie, Tibshirani & Wainwright (2015), Statistical Learning with Sparsity, which I find a very useful book (link broken right now - hopefully back up soon). Alternatively, regularize using the elastic net or using Bayesian approaches.

*Finally, much of the advice of not deleting insignificant predictors is given in a context of inferential statistics, and there it is valid. If your goal is to calculate p values, then these are invalidated if you first use the observations to simplify your model (regardless of the method you use), then use the same observations to check whether any of the remaining predictors are significant. This effect can be easily simulated, and I'd argue that anyone who does NHST should at least once simulate the effects of data dredging on data that follow the null hypothesis.
