Consider a multiple regression model $f$ (e.g. a random forest or neural network) where you try to predict a target $y$ given a two predictor variables $x_1$ and $x_2$:

$$f(x_1, x_2) = \hat y$$

Let's assume the relationship between $x_1$ and $y$ is quite simple, but its relationship with $x_2$ is more complicated. For example, let's assume the truth would be (where $\epsilon$ is some added noise): $$y = x_1 + \sin({x_2}^2) + \epsilon$$ In that case, is the following statement correct: Choosing a high variance (complex) model could capture the complicated relationship of $x_2$ and $y$, but it would overfit the simple relationship between $x_2$ and $y$?

And if that's true, what models allow different regularization for different inputs? I know for example that there are ridge regression implementations which allow to set a separate regularization parameter for every input. Are there others? How would it be done in a Random Forest or a Multi-Layer Perceptron?

  • 2
    $\begingroup$ I don't understand what you mean by "simple" and "complicated". After all, if you merely define $x_0=\sin(x_2^2)$, then $f(x_0,x_1)=x_0+x_1$ treats both variables equally. Could you define these concepts--preferably quantitatively--or at least provide a more detailed description? $\endgroup$
    – whuber
    Oct 4, 2017 at 22:00
  • $\begingroup$ Good question. Intuitively I would say complicated is something that is very non-linear. Something that can only be fit by a model with many degrees of freedom. $\endgroup$
    – Denwid
    Oct 4, 2017 at 22:10
  • $\begingroup$ I guess the additional step of introducing $x_0$ (doing a variable transformation) is exactly what makes the case more complicated. Let's assume no variable transformations are possible, the model has to learn the relationship itself. $\endgroup$
    – Denwid
    Oct 4, 2017 at 22:13

1 Answer 1


You have an estimate of $\hat{y}$ with a nonlinear function of $f(x_1,x_2)$, you perform a regression and penalize the parameter with predictor x_2. The way you have framed the problem it looks like a linear programming problem if you apply a linear relaxation to the function which has many solutions including a simplex algorithm

  • 1
    $\begingroup$ Linear programming involves a linear objective function with linear equality or inequality constraints. If you could formulate this as a mathematical programming problem it would be nonlinear programming problem and would not be solved by the simplex method. $\endgroup$ Nov 10, 2017 at 5:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.