# Beginner - Iteratively adding terms to regression model? [duplicate]

I'm learning about regression models via Andrew Ng's Coursera course. I have a question regarding automatically finding a good model.

Does it make sense (my guess is no) to iteratively add terms, or iteratively change a model to a different one, and then check the $$R^2$$ or MSE and see which one is best?

For example, if you have model like y = ax + b, and then you change it to $$y = ax^2 + b$$ and then $$y = ax^2 + ax + b$$, and keep doing so while continuously checking if it's better or worse than the other models.

Like I said, I'm just beginning learning this stuff, so I'm sure there's a better way to do this, but this just popped into my head as a possibility.

• Researching this a little further, I came across this thread math.stackexchange.com/questions/3069369/… which is basically asking the same thing but maybe a little clearer. In it AIC, BIC, and Chebychev polynomials are mentioned. So I may have my answer. If anyone can expand on these here that would be appreciated though. – isuckatprogramming May 2 at 16:01
• Another method I saw was choosing an arbitrarily large degree polynomial and using L2 regularization. This seems like it would introduce problems down the road though. – isuckatprogramming May 2 at 16:11

In the first step of analyses, we gather all possible features (including their exponents such as $$x_1^2$$ and interactions $$x_1x_2$$) which we believe would be useful in regression. Then, we choose a small subset among this possible feature set. A brute for approch would be training our model with all possible feature subsets. However, as you can guess, this is computationally very expensive. Instead, we prefer to find a good path through them. Forward-stepwise selection technique achieves this iteratively. First, you start with a single variable model with only one feature and an intercept. For this variable, you need to try out all features (train your model) you have gathered. Then, among them, you choose the 'best-performing' one. Here, the performance metric might be $$R^2$$, $$MSE$$ or anything else (including $$AIC, BIC$$). However, please do not forget to do this on an isolated data set (test set) which is not used in training phase.