How to apply feature engineering for classification efficiently using polynomials

Question

I have 21 numerical features and I want to build a classification model. Having tried a bunch of classifiers to the original features I would like to look at feature engineering.

One approach would be to look at multivariate polynomials. Instead of looking at $X_1$ and $X_2$ I look at features $X_1,X_2,X_1X_2,X_1^2$ and $X_2^2$. As I have 21 features this quickly grows.

What are statistical approaches that help me to decide a good degree of my polynomials?

As a remark: SVMs with polynomial kernel and multivariate adaptive splines already try some polynomials - right? It would be interesting to produce them myself.

Answer 1

Some things to consider:

• Saying your sample size is constant, then each additional feature n will reduce you generalization by log(n) factor -- you can find a good explanation for this here: http://www.visiondummy.com/2014/04/curse-dimensionality-affect-classification/.

• Saying that, you can use feature selection techniques(I like ones that are correlation based) to reduce number of multivariate features post creation.

• Using test + cross-validation with relevant classification score(sensitivity for example) and iterating on possible polynomial power.