it is also desirable that the activation function approximates the identity mapping near origin. To explain this, we should consider the activation of a neuron. Formally, the activation of a neuron is given by G (wx T + b) where G is the activation function. Usually, the weight vector w and bias b are initialized with values close to zero by the gradient descend method. Consequently, wx T + b will be close to zero. If G approximates the identity function near zero, its gradient will be approximately equal to its input. In other words, δ G ≈ wx T + b ⇐⇒ wx T + b ≈ 0. In terms of the gradient descend, it is a strong gradient which helps the training algorithm to converge faster.
https://en.wikipedia.org/wiki/Activation_function also quotes them saying:
Approximates identity near the origin: When activation functions have this property, the neural network will learn efficiently when its weights are initialized with small random values. When the activation function does not approximate identity near the origin, special care must be used when initializing the weights.
but I don't understand why the gradient approximates the input near zero. I think it should be a constant, maybe w. Can someone please explain this step by step?