Your criticism of logistic regression
Still, linear boundary, if used in something non-linear, what happens?
is based on a mischaracterization of the model.
While the functional form of the logistic regression model is often written
$$ f(x_1, x_2, \ldots, x_k) = \frac{1}{1 + \exp(- \sum_j \beta_j x_j )} $$
It would be more accurately, yet verbosely, written as this
$$ f(x_1, x_2, \ldots, x_k) = \frac{1}{1 + \exp(- \sum_j \beta_j \ f_j(x_1, x_2, \ldots, x_k))} $$
The prediction function is a linear combinations of arbitrary functions of the variables. This means it does not in general yield a linear decision boundary.
You also protest
but then you have to know these beforehand!
in reference to transformations of the predictors, but this is also false in general.
There are plenty of techniques used to incorporate determining the shape of non-linearities into the fitting of a logistic regression. It suffices to specify a general non-linear shape in terms of a basis expansion. For example, a basis of smoothing splines for one or multiple predictors can represent a large space of potential non-linear shapes, and there is also an appealing variational condition which characterizes them, in some sense, as the best you could possibly do.
Logistic regression, in a skilled practitioners hands, and when combined with meta-tools like the bootstrap, cross validation, and regularization, can be used to ask and answer more precise questions than an ad-hoc machine learning model (nothing against machine learning models, they are just a different sort of tool). It is much easier to take apart and look inside a logistic regression than a gradient boosted model, this allows for statistical inference and qualitative learnings.
Even the most powerful of machine learning type methods pay their respect to logistic regression in their choice of structure and loss function. A gradient boosted classifier can be expressed as:
$$ f(x_1, x_2, \ldots, x_k) = \frac{1}{1 + \exp(- \sum_j \beta_j \ T_j(x_1, x_2, \ldots, x_k)) } $$
where $T_j$ is a regression tree. The boosted model is fit to the same loss function as logistic regression, so the only change in structure is in the choice of basis expansion. It's quite reasonable to think of a gradient boosted classifier as just a logistic regression with trees as basis functions, and a greedy algorithm to choose the successive basis elements.
