Why are activation functions needed in neural networks? I know that it is to capture "non-linearities", but I have never been able to find a proper down-to-earth explanation. In particular, I am curious about the ReLU function as it is quite effective, but seems like such a simple function. How does that capture non-linearities?
In general, non-linearities are used to flexibly squash the input through a function and pave the way recovering a higher-level abstraction structure, but allow me to become more specific.
One of the most informative illustrations I have found about ReLU activations is the following:
The picture is from the work of A. Punjani and P. Abbeel, and depicts a simple neural network with a ReLU activation unit. Now imagine you have a 2D input space, as it can be seen the ReLU unit $\phi$, actually partitions the input space and regresses towards your desired outcome, where the space partitioning is a non-linear operation. That's the reason it is so powerful as it allows combinations of different input transformations, which are learnt from the dataset itself.
More specifically, the author's description is:
A pictorial representation of the flexibility of the ReLU Network Model. Each hidden unit can be thought of as defining a hyperplane (line) in the 2D input space pictured. The data points (grey) each fall somewhere in input space and each has a value we wish to regress (not pictured). Consider the hidden unit $i$, drawn in purple. The purple arrow points in the direction of weight vector $Wi$ , and has length according to $Bi$ . Points on one side of this line do not activate unit $i$, while points on the other side (shaded) cause positive output. This effectively partitions the input space, and the partitions generated by considering many hidden units (blue) together split the space into regions. These regions give the model flexibility to capture structure in the input data. In the ReLU Network Model, the partitions are learned from data.