I have 8 predictors and for 4 of them, the AV plots have a slope close to 0. Is that enough information for me to leave out these predictors in my model?
If there's not otherwise pattern (such as a nonlinear trend), lack of slope in an added-variable plot would tend to indicate that the variable wouldn't help you explain the response over what was in the model.
However, note that this is still a form of model selection, and will have a similar effect on your inferences* that other forms of stepwise selection would have.
* (on test significance levels, confidence interval coverage, bias of estimates and standard errors and so on)