# Do models overfit simple predictors alongside complicated predictors?

Consider a multiple regression model $f$ (e.g. a random forest or neural network) where you try to predict a target $y$ given a two predictor variables $x_1$ and $x_2$:

$$f(x_1, x_2) = \hat y$$

Let's assume the relationship between $x_1$ and $y$ is quite simple, but its relationship with $x_2$ is more complicated. For example, let's assume the truth would be (where $\epsilon$ is some added noise): $$y = x_1 + \sin({x_2}^2) + \epsilon$$ In that case, is the following statement correct: Choosing a high variance (complex) model could capture the complicated relationship of $x_2$ and $y$, but it would overfit the simple relationship between $x_2$ and $y$?

And if that's true, what models allow different regularization for different inputs? I know for example that there are ridge regression implementations which allow to set a separate regularization parameter for every input. Are there others? How would it be done in a Random Forest or a Multi-Layer Perceptron?

• I don't understand what you mean by "simple" and "complicated". After all, if you merely define $x_0=\sin(x_2^2)$, then $f(x_0,x_1)=x_0+x_1$ treats both variables equally. Could you define these concepts--preferably quantitatively--or at least provide a more detailed description?
– whuber
Oct 4, 2017 at 22:00
• Good question. Intuitively I would say complicated is something that is very non-linear. Something that can only be fit by a model with many degrees of freedom. Oct 4, 2017 at 22:10
• I guess the additional step of introducing $x_0$ (doing a variable transformation) is exactly what makes the case more complicated. Let's assume no variable transformations are possible, the model has to learn the relationship itself. Oct 4, 2017 at 22:13

You have an estimate of $\hat{y}$ with a nonlinear function of $f(x_1,x_2)$, you perform a regression and penalize the parameter with predictor x_2. The way you have framed the problem it looks like a linear programming problem if you apply a linear relaxation to the function which has many solutions including a simplex algorithm