How to add features of higher order for Logistic Regression I'm starting my first Machine Learning project to classify some entities and I decided to use Logistic Regression for the task.
Initially I starter with around 10 features and I can see that my model is underfitting the data (F-Score around 0.63). 
That can be explained because all of my features are of first order and so my hypothesis is a first order polynomial. 
I would like to add more of higher order features, but I quickly realized that I don't have a good intuition on how to do that. I could take each of my features $X_n$ and add new ones $X_{n^2}$, $X_{n^3}$ etc. I could also start adding more complex features like $X_1$ * $X_2$ etc. 
Immediatelly I noticed that there are countless possibilities. How do I start? What are good practices in adding more features. How can I avoid overfitting the data?
 A: If you are really want to create higher order features to a logistic regressor then I would suggest you expand your features with interaction  between features $X_1*X_2$, nonlinear features like $log(X_1)$ and $X_1^2$. Everything exactly like you proposed.
Finally to avoid over-fitting and at the same time doing variable selection apply a LASSO regularizer, it will both penalize model complexy and also induce sparsity. Only the subset of features, high order features that are of higher importance will be kept by the model.
You might also want to consider non linear models, they try to discover the optimal non-linearity by themselves (e.g. neural networks).
A: First of all, it's important to know the number of entites you have.
The number of regressors you can have will be very dependent to that.
Have you split your data in a training set and a validation set ?
After, you're not necessarily "under fitting", maybe a model with a F-score of 0.63 is the best model possible. 
Be careful to not add to many features, it will add variance to your model. To know which feature you have to keep, you have to use a significance test for every feature. 
You can see here a example on R : http://www.r-tutor.com/elementary-statistics/logistic-regression/significance-test-logistic-regression
If there insignificant feature, you delete one by one the features with the higher p_value.
You will quickly see that the feature with a high polynomial degree are often insignificant.
A: Your best guess to 'where to stop' is to continuously plot your metrics (precision/recall/accuracy/misclassification) with your test set. As soon as they start to deteriorate you're likely overfitting and might want to reduce the number of features. Also, you have to use your intuition while selecting which polinomial features to add. Prioritize the features that seem to be more relevant.
See the error-analysis section here: http://www.holehouse.org/mlclass/11_Machine_Learning_System_Design.html
Also, another way to add features that might improve the algo is to breakdown factor/categorical features into dummy features (https://discuss.analyticsvidhya.com/t/how-to-handle-categorical-variables-in-logistic-regression/247/4)
Hope this helps
