How does nonlinearity in neural networks find meaningful features?

I'm currently reading through the book 'Neural Network Methods for Natural Language Processing' by Goldberg and I'm confused with the following statement:

The nonlinearity of the classifier, as defined by the network structure, is expected to take care of finding the indicative feature combinations, alleviating the need for feature combination engineering.

In my understanding, the activation function in each node in a nn brings in the non-linearity because the function itself is non-linear (e.g. sigmoid, relu, etc.) and the weights for each input dimension that are multiplied by the input of each node are changed based on the result they provide. But I don't understand how this does the 'feature engineering' for us, i.e. how it determines which feature combinations are meaningful for, e.g. a classification task?

Each layer in your neural net multiples your inputs with a weight matrix and adds a bias term, i.e. output of the first layer is $$W_1x+b_1$$, and if you don't have a nonlinear activation function, the second layer output will be $$W_2(W_1x+b_1)+b_2=$$ $$W_2W_1x$$ $$+W_2b_1+b_2$$, which is still in $$Wx+b$$ form. It doesn't matter how many layer you add, output will be still in $$Wx+b$$ form. For the sake of simplicity assume one neuron in output layer, i.e. binary classification. You will decide based on $$Wx+b<\theta$$, which means, you'll decide based on $$w_1x_1+...w_nx_n<\theta$$. There are no interaction terms, no polynomial features etc. We still use the same set of features, even though you had thousands of layers. But, even if you had applied a very simple nonlinear function like $$f(x)=x^2$$, you'd have terms like $$x_i^2$$, $$x_ix_j$$ in your decision surface, which signals the capability of creating and using new feature combinations.
• Different nonlinearities tend to generate different feature interactions. I've used $f(x)=x^2$ just to simplify the way we think. ReLU seems like a very simple function but it can be used to construct an estimate of $f(x)=ax^2\approx ReLU(x)+ReLU(-x)$ around $x=0$ for example. Or you write tanh and sigmoid functions via Taylor expansion and convince yourself to the idea that you actually use polynomial features inherently. – gunes Jan 2 at 14:05