ReLUs don't suffer from vanishing gradients, but they have their own issues. The main issue for ReLU is the dead-ReLU phenomenon, i.e. a neuron whose activation becomes zero for all samples due to ReLU. This neuron is irrecoverable, because gradients through it are always zero. Leaky ReLU and other activation functions were created to deal with this issue.


No because $a^2 = [(-1)(-a)]^2 = [(-1)^2(-a)^2] = (-a)^2$. For the gradient, you'd have $2(-a)\left[\frac{\partial}{\partial \theta}(-a) \right] = 2a \left[\frac{\partial}{\partial \theta} a \right]$ due to the chain rule.


Hope the following text from Wikipedia may dissipate your concerns: A number of methods have been proposed to accelerate the sometimes slow convergence of the EM algorithm, such as those using conjugate gradient and modified Newton's methods (Newton–Raphson).[25] Also, EM can be used with constrained estimation methods. Parameter-expanded expectation ...

Only top voted, non community-wiki answers of a minimum length are eligible