<ReLU>
Pros:
- It mitigates Vanishing Gradient Problem.
- It's not computationally expensive.
Cons:
- It causes Dying ReLU Problem.
- It's non-differentiable at
x=0.
<Sigmoid>
Pros:
- It normalises input values so the convergence is more stable than ReLU.
- It mitigates Exploding Gradient Problem.
- It avoids Dying ReLU Problem.
- It's continuously differentiable.
Cons:
- It causes Vanishing Gradient Problem.
- It's computationally expensive because of exponential operation.