Should I consider using nonlinear expression in logistic regression when the training set is just nonlinear? Conventional logistic regression expresses log odds as a linear expression of predictors, i.e., $\beta'x$. And the problem reduces to using linear classifier $\beta'x=0$ to classify. But sometimes the training set is apparently nonlinear (may be tested using linear programming), for example, if I even visualize two almost separable circles are the two classes in a two-dimensional training set, should I just use something like $\beta_1x_1^2+\beta_2x_2^2+\beta_3$ to express log odds? Is this arbitrary enough since the functional form is based on my visualization? But on the other hand, isn't conventional logistic regression also quite arbitrary since it visualizes linearity using techniques such as linear programming? 

Do people in practice attempt to fit in any nonlinear functions for logistic regression?
 A: The term "linear" here refers to the relationship between the features (predictors) and the parameters in the regression equation
$$\mathbb{E}[\,y\mid \mathbf{f}\,]=\mathbf{f}^T\boldsymbol{\beta}$$
Here $\boldsymbol{\beta}=[\beta_1,\ldots,\beta_m]$ is the vector of parameters to be estimated from the data, while for a given data point $(\mathbf{x},y)$, the vector $\mathbf{f}(\mathbf{x})=[f_1,\ldots,f_m]$ is a set of features computed from $\mathbf{x}$, and for logistic regression, $y=\mathrm{logit}[\Pr(\mathrm{class}=1)]$.
In your example, the feature-mapping
$$\mathbf{f}(\mathbf{x})=[x_1^2,x_2^2,1]$$
is nonlinear, but the regression is still linear in the parameters $\boldsymbol{\beta}$. 

For your example problem of "separating circles", you can see a nice demonstration of linear vs. nonlinear approaches using the TensorFlow Playground site:


*

*Using a linear model with linear features $(x,y)$, the data cannot be separated.

*The data can be separated using a linear model with nonlinear features $(x^2,y^2)$.

*The data can be separated using linear features, but only using a nonlinear model.
(For each example above, click on the link and then press "play" to train the classifier.)
The first example can do no better than chance (i.e. 50% error). The second example is the one from your question. The third example shows a neural network with a hidden layer that learns 3 new "hybrid features", which are then used to classify the data.

For your second question, I would say it is quite common to use nonlinear feature mappings. This is often done in the context of the so-called "kernel trick" (e.g. for SVMs). Classically, these feature mappings are pre-specified. As shown in the third example above, nonlinear feature mappings can also be learned, which is often done in the context of deep learning.
