ReLU:
- 's pros:
- It mitigates Vanishing Gradient Problem.
- It's not computationally expensive.
- 's cons:
- It causes Dying ReLU Problem.
- It's non-differentiable at
x=0.
Sigmoid:
- 's pros:
- It normalises input values so the convergence is more stable than ReLU.
- It mitigates Exploding Gradient Problem.
- It avoids Dying ReLU Problem.
- It's continuously differentiable.
- 's cons:
- It causes Vanishing Gradient Problem.
- It's computationally expensive because of exponential operation.