1
$\begingroup$

As far as I know, $y=\beta x$ is a not a complex model since we have a polynomial of the first order for all variables $x_i$. I am studying the linear the bias variance trade-off, and the lecture mentioned that we can reduce the complexity of the model by reducing the number of variables. This made me confused with my initial intuition about the definition of the complexity of the model. Can any one explain to me how the number of variable can make the above model more complex and how it could go through all the data points while it is only a first grade model? Thanks

$\endgroup$
3
  • $\begingroup$ The general idea is that you want your model to has a few variables/terms as possible (principle of parsimony). The fewer terms you have, the easier it is for someone to interpret your model. You're also right in your thinking by the way - adding polynomial terms higher than degree one leads to an increase in model complexity. In short, model complexity increases as you add more variables, or variables with some sort of transformation ($x^n$, $\log(x)$, etc.). $\endgroup$
    – ralph
    Apr 24, 2022 at 5:34
  • $\begingroup$ @ralph Would you want to post that as an answer? $\endgroup$
    – Dave
    Apr 24, 2022 at 5:54
  • $\begingroup$ Parsimony is the enemy of predictive discrimination. Parsimony is not how the world works. And the data do not possess sufficient information content to allow one to determine which variables are the "big predictors". $\endgroup$ Apr 24, 2022 at 12:05

1 Answer 1

1
$\begingroup$

The general idea is that you want your model to have a few variables/terms as possible (principle of parsimony). The fewer terms you have, the easier it is for someone to interpret your model.

You're also right in your thinking by the way - adding polynomial terms higher than degree one also leads to an increase in model complexity.

In short, model complexity increases as you add more variables, or variables with some sort of transformation ($x^n$, $\log(x)$, etc.).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.