Why are activation functions needed in neural networks? [duplicate]

Why are activation functions needed in neural networks? I know that it is to capture "non-linearities", but I have never been able to find a proper down-to-earth explanation. In particular, I am curious about the ReLU function as it is quite effective, but seems like such a simple function. How does that capture non-linearities?

• @Sycorax, how can this be duplicate of another question that's asked more than 2 years after this? Also, the two questions answer slightly different things. – pir Aug 26 '19 at 2:07
• Duplicates don’t have to be temporally ordered. – Sycorax Aug 26 '19 at 2:35

The picture is from the work of A. Punjani and P. Abbeel, and depicts a simple neural network with a ReLU activation unit. Now imagine you have a 2D input space, as it can be seen the ReLU unit $\phi$, actually partitions the input space and regresses towards your desired outcome, where the space partitioning is a non-linear operation. That's the reason it is so powerful as it allows combinations of different input transformations, which are learnt from the dataset itself.
A pictorial representation of the flexibility of the ReLU Network Model. Each hidden unit can be thought of as defining a hyperplane (line) in the 2D input space pictured. The data points (grey) each fall somewhere in input space and each has a value we wish to regress (not pictured). Consider the hidden unit $i$, drawn in purple. The purple arrow points in the direction of weight vector $Wi$ , and has length according to $Bi$ . Points on one side of this line do not activate unit $i$, while points on the other side (shaded) cause positive output. This effectively partitions the input space, and the partitions generated by considering many hidden units (blue) together split the space into regions. These regions give the model flexibility to capture structure in the input data. In the ReLU Network Model, the partitions are learned from data.