When to drop a term from a regression model? Could anyone advise if the following makes sense:
I am dealing with an ordinary linear model with 4 predictors. I am in two minds whether to drop the least significant term. It's $p$-value is a little over 0.05. I have argued in favor of dropping it along these lines:
Multiplying the estimate of this term by (for example) the interquartile range of the sample data for this variable, gives some meaning to the clinical effect that keeping this term has on the overall model. Since this number is very low, approximately equal to typical intra-day range of values that the variable can take when measuring it in a clinical setting, I see it as not clinically significant and could therefore be dropped to give a more parsimonious model, even though dropping it reduces the adjusted $R^2$ a little.
 A: The most common advice these days is to get the AIC of the two models and take the one with the lower AIC.  So, if your full model has an AIC of -20 and the model without the weakest predictor has an AIC > -20 then you keep the full model.  Some might argue that if the difference < 3 you keep the simpler one.  I prefer the advice that you could use the BIC to break "ties" when the AIC's are within 3 of each other.
If you're using R then the command to get the AIC is... AIC.
I do have a textbook on modelling here from the early 90s suggesting that you drop all of your predictors that are not significant.  However, this really means you'll drop independent of the complexity the predictor adds or subtracts from the model.  It's also only for ANOVA where significance is about variability explained rather than the magnitude of the slope in light of what other things have been explained.  The more modern advice of using AIC takes these factors into consideration.  There's all kinds of reasons the non-significant predictor should be included even if it's not significant.  For example, there may be correlation issues with other predictors to it may be a relatively simple predictor.  If you want the simplest advice go with AIC and use BIC to break ties and use a difference of 3 as your window of equality.  Otherwise, provide many many more details about the model and you could get more specific advice for your situation.
A: There are at least two other possible reasons for keeping a variable:
1) It affects the parameters for OTHER variables.
2) The fact that it is small is clinically interesting in itself
To see about 1, you can look at the predicted values for each person from a model with and without the variable in the model.  I suggest making a scatterplot of these two sets of values. If there are no big differences, then that's an argument against this reason
For 2, think about why you had this variable in the list of possible variables. Is it based on theory? Did other research find a large effect size? 
A: What are you using this model for?  Is parsimony an important goal?
More parsimonious models are preferred in some situations, but I wouldn't say parsimony is a good thing in itself.  Parsimonious models can be understood and communicated more easily, and parsimony can help guard against over-fitting, but often times these issues are not major concerns or can be addressed another way.  
Approaching from the opposite direction, including an extra term in a regression equation has some benefits even in situations in which the extra term itself isn't of interest and it doesn't improve the model fit much... you may not think that it is an important variable to control for, but others might. Of course, there are other very important substantive reasons to exclude a variable, e.g. it might be caused by the outcome.
A: From your wording it sounds as if you're inclined to drop the last predictor because its predictive value is low; a substantial change on that predictor would not imply a substantial change on the response variable.  If that is the case, then i like  this criterion for including/dropping the predictor.  It's more grounded in practical reality than the AIC or BIC can be, and more explainable to your audience for this research.
A: I have never understood the wish for parsimony.  Seeking parsimony destroys all aspects of statistical inference (bias of regression coefficients, standard errors, confidence intervals, P-values).  A good reason to keep variables is that this preserves the accuracy of confidence intervals and other quantities.  Think of it this way: there have only been developed two unbiased estimators of residual variance in ordinary multiple regression: (1) the estimate from the pre-specified (big) model, and (2) the estimate from a reduced model substituting generalized degrees of freedom (GDF) for apparent (reduced) regression degrees of freedom.  GDF will be much closer to the number of candidate parameters than to the number of final "significant" parameters.
Here's another way to think of it.  Suppose you were doing an ANOVA to compare 5 treatments, getting a 4 d.f. F-test.  Then for some reason you look at pairwise differences between treatments using t-tests and decided to combine or remove some of the treatments (this is the same as doing stepwise selection using P, AIC, BIC, Cp on the 4 dummy variables).  The resulting F-test with 1, 2, or 3 d.f. will have inflated type I error.  The original F-test with 4 d.f. contained a perfect multiplicity adjustment.
A: These answers about selection of variables all assume that the cost of the observation of variables is 0.
And that is not true. 
While the issue of selection of variables for a given model may or may not involve selection, the implications for future behavior DOES involve selection. 
Consider the problem of predicting which college lineman will do best in the NFL. You are a scout. You must consider which qualities of the current linemen in the NFL are most predictive of their success. You measure 500 quantities, and begin the task of the selection of the quantities which will be needed in the future. 
What should you do? Should you retain all 500? Should some (astrological sign, day of the week born on) be eliminated?
This is an important question, and is not academic. There is a cost to the observation of data, and the framework of cost-effectiveness suggests that some variables NEED NOT be observed in the future, since their value is low.
