1
$\begingroup$

It is known that increasing the complexity of a regression or classification model reduces the training error ( see e.g. Elements of Statistical Learning Chapter 2 ).

My question is whether the converse of this is guaranteed to be true?

Does reducing the number of explanatory variables guarantee an increase in training error?

This will have practical implications for model building, i.e. if the training error is not acceptable with $k$ explanatory variables, then it is pointless looking for a better model with $<k$ explanatory variables.

At first thought it seems 'obvious' it must be true but are there any situations where it might not be true? For example if one thinks of collinearity in the predictors, which if present reduces their accuracy, it raises the question if removing some of them to eliminate the collinearity, then with increased accuracy of the coefficients it may lead to reduced training error? I suspect that this cannot be true but I do not know how to explain it convincingly.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

"It is known that increasing the complexity of a regression or classification model reduces the training error". So... it is trivial!

Assume model A is more complex than model B. Then A shows less training error than B

Now assume instead A is simpler than B. Therefore B is more complex than A. We conclude that shows less training error than B

Improve this answer
$\endgroup$
1
  • $\begingroup$ There's something missing from your last sentence. Also, I wonder if you accidentally switched the letters there? $\endgroup$
    – gung - Reinstate Monica
    Commented Apr 2, 2019 at 14:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.