1
$\begingroup$

When model complexity goes up, why test error also goes up, instead of staying on a similar level ?

I feel this is countering the intuition that when you add random parameters to a model, it should not do harmful if not doing good.

Please help me understand. Thanks.

$\endgroup$

1 Answer 1

4
$\begingroup$

When you increase complexity of your model, it is more likely to overfit, meaning it will adapt to training data very well, but will not figure out general relationships in the data. In such case, performance on a test set is going to be poor. Such model is great at remembering, but when it encounters data it has not seen before, it gets 'confused', if you like.

An example of that is regression with polynominals. When you have data correlated linearly, the best model is polynominal of 1st degree. You can use more complex model, like polynominal of degree equal to size of your training set. Your model will be great at memorizing a training set, but will most likely fail to model a linear relationship properly where a 1st degree polynominal does a great job. This leads to poor test set performance.

$\endgroup$
2
  • $\begingroup$ Following the arguments in the answer, can I say that as the model complexity goes up, the bias in the test set also goes up? $\endgroup$
    – vtshen
    Commented Aug 30, 2019 at 19:02
  • $\begingroup$ Bias is a feature of a model not a test set. Having said that, no. Highly complex models tend to have high variance and low bias. $\endgroup$
    – Marcin
    Commented Aug 30, 2019 at 20:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.