Is RSS decreasing or non-increasing I am currently reading The Elements of Statistical Learning and on the section on best subset selection in the chapter in linear regression it says that as we add predictors to the model the RSS is always decreasing. Does this mean to say non-increasing? I can see no reason why it should always be decreasing?
Thanks for any help
 A: Technically, it is non-increasing. Suppose your RSS was already 0, or that you added a vector of all zeros as a predictor (or an exact duplicate of a previous vector, or a linear combination of previous vectors for linear models, and so on).
Practically, you won't already have an RSS of 0, and so there's some remaining vector of residuals. For every new vector that is independent of previous vectors that you add, so long as it's not totally orthogonal to the residuals, you'll be able to further explain your training data and drive the RSS lower. Because this will happen for any non-pathological data, adding random values as predictors will work!
Why is it true that increasing is ruled out? Because your model is an optimization model. If using the new data would make the model worse (increase the RSS) then parameters are chosen such that the new data is not used (say, by multiplying it by 0).
A: As you add predictors in your model RSS decreasing this means that adequacy of model is improved for training data(but not for test data) and this is not over target to add more and more predictor in over model 
