ReLU:

• 's pros:
  • It mitigates Vanishing Gradient Problem.
  • It's not computationally expensive.
• 's cons:
  • It causes Dying ReLU Problem.
  • It's non-differentiable at x=0.

Sigmoid:

• 's pros:
  • It normalises input values so the convergence is more stable than ReLU.
  • It mitigates Exploding Gradient Problem.
  • It avoids Dying ReLU Problem.
  • It's continuously differentiable.
• 's cons:
  • It causes Vanishing Gradient Problem.
  • It's computationally expensive because of exponential operation.

<ReLU>

Pros:

• It mitigates Vanishing Gradient Problem.
• It's not computationally expensive.

Cons:

• It causes Dying ReLU Problem.
• It's non-differentiable at x=0.

<Sigmoid>

Pros:

• It normalises input values so the convergence is more stable than ReLU.
• It mitigates Exploding Gradient Problem.
• It avoids Dying ReLU Problem.
• It's continuously differentiable.

Cons:

• It causes Vanishing Gradient Problem.
• It's computationally expensive because of exponential operation.