I have 21 numerical features and I want to build a classification model. Having tried a bunch of classifiers to the original features I would like to look at feature engineering.

One approach would be to look at multivariate polynomials. Instead of looking at $X_1$ and $X_2$ I look at features $X_1,X_2,X_1X_2,X_1^2$ and $X_2^2$. As I have 21 features this quickly grows.

What are statistical approaches that help me to decide a good degree of my polynomials?

As a remark: SVMs with polynomial kernel and multivariate adaptive splines already try some polynomials - right? It would be interesting to produce them myself.


Some things to consider:

  • Saying your sample size is constant, then each additional feature n will reduce you generalization by log(n) factor -- you can find a good explanation for this here: http://www.visiondummy.com/2014/04/curse-dimensionality-affect-classification/.

  • Saying that, you can use feature selection techniques(I like ones that are correlation based) to reduce number of multivariate features post creation.

  • Using test + cross-validation with relevant classification score(sensitivity for example) and iterating on possible polynomial power.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.