The answer provided by Demetri already provides a sufficient answer here (+1). If your model's intent is to predict, then the $p$-values matter a lot less than the model's ability to guess on future outcomes. As he mentioned, tools like cross-validation go a long way to determining whether or not your model is really good at prediction. To a degree, other inferential data like your standard errors and the magnitude of the associations are probably much better barometers of your model's ability to predict anyway.
If the goal is theory testing (the major part of my answer here), then adding or removing variables based on $p$ values is a poor practice and entirely misses the objective of the analysis. Here I will use a practical example from my own field to perhaps demonstrate why variable selection based on $p$ values is a bad idea.
Suppose we are interested in two skills which predict reading. The first skill is phonological awareness (PA), or the ability to recognize the sound rules in a word (e.g. knowing that "fight" and "might" rhyme). The second is morphological awareness (MA), which is the ability to recognize the morphological constructions of a word (e.g. "eyeball" is separately made up of the words "eye" and "ball"). It may be the case that in the population, the relationship between reading and these two cognitive skills is:
$$
\text{Reading} = 1 + (2 \times \text{PA}) + (.0001 \times \text{MA}) + \epsilon
$$
I'm writing these in a rush, so these wouldn't reflect their actual values in my field. We can anyway easily see that PA has a strong relationship with reading, whereas MA has a poor relationship. But perhaps our inference based on past research is that these predictors should have an equally strong relationship, perhaps something like:
$$
\text{Reading} = 1 + (2 \times \text{PA}) + (2 \times \text{MA}) + \epsilon
$$
It may be the case that we find some data to test how important each association is. We can simulate the true population values in R and see what the regression tells us:
#### Simulate ####
set.seed(1234)
n <- 1000
pa <- rnorm(n)
ma <- rnorm(n)
read <- 1 + 2*pa + (.0001 * ma) + rnorm(n)
df <- data.frame(pa,ma,read)
pairs(df)
#### Fit ####
fit <- lm(read ~ pa + ma)
summary(fit)
The model predictably shows that $X_1$ has a strong association and $X_2$ doesnt:
Call:
lm(formula = read ~ pa + ma)
Residuals:
Min 1Q Median 3Q Max
-3.13161 -0.71957 0.03478 0.70215 3.05316
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.02996 0.03204 32.150 <2e-16 ***
pa 2.01770 0.03217 62.712 <2e-16 ***
ma -0.03680 0.03270 -1.125 0.261
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 1.013 on 997 degrees of freedom
Multiple R-squared: 0.798, Adjusted R-squared: 0.7976
F-statistic: 1969 on 2 and 997 DF, p-value: < 2.2e-16
Well we already know that this is how it should look (which is never the case in reality). Our model is nudging us to answer the question of how important each variable is. You can see that it does pretty good at capturing the intercept and the slope for PA while getting it a bit wrong with the negative coefficient for MA, but regardless showing that it is quite weak (and so unsurprisingly the $p$ values are not statistically significant even with $n=1000$ observations). That is the entire point of a regression. We are sampling from the population and trying to use that sample to guess what the population is. By tossing out the variables we cared about, we destroy our inference about the variables in question, in essence throwing our our assumptions that each slope $\beta = 2$ (though in a situation like this, its probably better to go full Bayesian anyway given we have prior information about what the effect should be, but that is a separate point).
All of this is particularly germane to practical implications of the model. If one skill happens to be great at predicting reading while the other doesn't, then naturally our model with a poorly predictive cognitive skill is extremely important for informing educators to not waste their time and resources on this issue. How can others know this if you don't include it in your model? And yet this seems to happen more commonly than one would think.
