I'll add another "yes". I've always been taught -- and I've tried to pass it along -- that the primary consideration in covariate choice is domain knowledge, not statistics. In biostatistics, for instance, if I'm modelling some health outcome on individuals, then no matter what the regression says, you'll need some darn good arguments for me not to include age, race, and sex in the model.

It also depends on the purpose of your model. If the purpose is gaining better understanding of what factors are most associated with your outcome, then building a parsimonious model has some virtues. If you care about prediction, and not so much about understanding, then eliminating covariates may be a smaller concern.

(Finally, if you're planning to use statistics for variable selection, check out what Frank Harrell has to say on the subject -- http://www.stata.com/support/faqs/statistics/stepwise-regression-problems/, and his book Regression Modeling Strategies. Briefly, by the time you've used stepwise or similar statistically-based strategies for choosing the best predictors, then any tests of "are these good predictors?" are terribly biased -- of course they're good predictors, you've chosen them on that basis, and so the p values for those predictors are falsely low.)