Sometimes we want to use some features in our original dataset to create polynomial features in order to add non-linearity to our model.

The question is how to choose those features? Do we choose features with a relatively high correlation with the target variable? Do we choose features that have high importance according to some model that provides a list of feature importances like random forest or Xgboost? Or what?

  • $\begingroup$ Does anyone know what to do if feature x1*x2 and feature x2*x3 are highly correlated? $\endgroup$ – Luis Pinto Feb 28 '20 at 0:24

I wish there had been an easy recipe, but unfortunately no. The alternatives you listed are among the many intuitive heuristics for feature selection, e.g. random forest / xgboost feature importances, lasso coefficients, logistic regression weights, correlation coefficients etc. You need to do model selection after having some set of polynomial features.

By the way, usage of single variable polynomial features in decision tree based algorithms sometimes might not have an impact on your performance because these transformations do not change the total ordering of the variables if odd-powered and therefore decision boundaries might be similar, i.e. $x_1<x_2 \rightarrow x_1^3<x_2^3$, so the boundary will be still between the two.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.