I'm currently reading through the book 'Neural Network Methods for Natural Language Processing' by Goldberg and I'm confused with the following statement:

The nonlinearity of the classifier, as defined by the network structure, is expected to take care of finding the indicative feature combinations, alleviating the need for feature combination engineering.

In my understanding, the activation function in each node in a nn brings in the non-linearity because the function itself is non-linear (e.g. sigmoid, relu, etc.) and the weights for each input dimension that are multiplied by the input of each node are changed based on the result they provide. But I don't understand how this does the 'feature engineering' for us, i.e. how it determines which feature combinations are meaningful for, e.g. a classification task?


1 Answer 1


Each layer in your neural net multiples your inputs with a weight matrix and adds a bias term, i.e. output of the first layer is $W_1x+b_1$, and if you don't have a nonlinear activation function, the second layer output will be $W_2(W_1x+b_1)+b_2=$ $W_2W_1x$ $+W_2b_1+b_2$, which is still in $Wx+b$ form. It doesn't matter how many layer you add, output will be still in $Wx+b$ form. For the sake of simplicity assume one neuron in output layer, i.e. binary classification. You will decide based on $Wx+b<\theta$, which means, you'll decide based on $w_1x_1+...w_nx_n<\theta$. There are no interaction terms, no polynomial features etc. We still use the same set of features, even though you had thousands of layers. But, even if you had applied a very simple nonlinear function like $f(x)=x^2$, you'd have terms like $x_i^2$, $x_ix_j$ in your decision surface, which signals the capability of creating and using new feature combinations.

So, repeated layers with in-between non-linear activations inherently creates feature combinations. In the output layer, you see these newly engineered features as inputs to your neurons.

  • $\begingroup$ I've read more into the topic since asking the question and got a better idea of how it helps. However, this answer sums it up really nicely and certainly helped, thank you! $\endgroup$ Jan 1, 2019 at 19:08
  • $\begingroup$ To add on the answer: Does the type of nonlinearity matter? You mentioned a quadratic function in your answer but mostly something like tanh or ReLu is used from what I've seen, which in the case of ReLu is essentially just a clipping function. Does that the same as e.g. a quadratic function? $\endgroup$ Jan 2, 2019 at 13:32
  • 2
    $\begingroup$ Different nonlinearities tend to generate different feature interactions. I've used $f(x)=x^2$ just to simplify the way we think. ReLU seems like a very simple function but it can be used to construct an estimate of $f(x)=ax^2\approx ReLU(x)+ReLU(-x)$ around $x=0$ for example. Or you write tanh and sigmoid functions via Taylor expansion and convince yourself to the idea that you actually use polynomial features inherently. $\endgroup$
    – gunes
    Jan 2, 2019 at 14:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.