Why is ReLU so popular despite being NOT zero-centered Activation functions should be zero-centered (Reference) and that's why tanh is preferred over sigmoid.
But ReLU is not zero centered and still is often the first choice. I know it solves the issue of gradient saturation in sigmoid and tanh.
My question is why is ReLU NOT being zero-centered an issue?
 A: ReLU's non-zero centering is an issue. ReLUs are popular because it is simple and fast. On the other hand, if the only problem you're finding with ReLU is that the optimization is slow, training the network longer is a reasonable solution.
However, it's more common for state-of-the-art papers to use more complex activations. A general strategy is to come up with a function that retains approximately the identity function for positive values, but also controls the means and variances. For example, the mish activation has achieved state-of-the-art results recently.
But maybe you face time or cost constraints, or other problems with ReLU (e.g. dead units). In these cases, you may  be interested in one of these alternative activations.

*

*Leaky ReLUs have a negative portion, and this may reduce the occurrence of the detrimental "zig zag" effect that is noted in the linked thread. This paragraph is not at all conclusive, but I haven't found a better paper. (If you've found a better paper about the optimization dynamics of Leaky ReLU units, please share it in comments!)

Andrew L. Maas, Awni Y. Hannun, Andrew Y. Ng "Rectifier Nonlinearities Improve Neural Network Acoustic Models"

The choice of rectifier function used in the DNN appears unimportant for both frame-wise and WER metrics. Both the leaky and standard ReL networks perform similarly, suggesting the leaky rectifiers’ non-zero
gradient does not substantially impact training optimization. During training we observed leaky rectifier
DNNs converge slightly faster, which is perhaps due
to the difference in gradient among the two rectifiers.


*

*Djork-Arné Clevert, Thomas Unterthiner, Sepp Hochreiter. "Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)"


We introduce the "exponential linear unit" (ELU) which speeds up learning in deep neural networks and leads to higher classification accuracies. Like rectified linear units (ReLUs), leaky ReLUs (LReLUs) and parametrized ReLUs (PReLUs), ELUs alleviate the vanishing gradient problem via the identity for positive values. However, ELUs have improved learning characteristics compared to the units with other activation functions. In contrast to ReLUs, ELUs have negative values which allows them to push mean unit activations closer to zero like batch normalization but with lower computational complexity. Mean shifts toward zero speed up learning by bringing the normal gradient closer to the unit natural gradient because of a reduced bias shift effect. While LReLUs and PReLUs have negative values, too, they do not ensure a noise-robust deactivation state. ELUs saturate to a negative value with smaller inputs and thereby decrease the forward propagated variation and information. Therefore, ELUs code the degree of presence of particular phenomena in the input, while they do not quantitatively model the degree of their absence. In experiments, ELUs lead not only to faster learning, but also to significantly better generalization performance than ReLUs and LReLUs on networks with more than 5 layers. On CIFAR-100 ELUs networks significantly outperform ReLU networks with batch normalization while batch normalization does not improve ELU networks. ELU networks are among the top 10 reported CIFAR-10 results and yield the best published result on CIFAR-100, without resorting to multi-view evaluation or model averaging. On ImageNet, ELU networks considerably speed up learning compared to a ReLU network with the same architecture, obtaining less than 10% classification error for a single crop, single model network.


*

*Günter Klambauer, Thomas Unterthiner, Andreas Mayr, Sepp Hochreiter "Self-Normalizing Neural Networks"


Deep Learning has revolutionized vision via convolutional neural networks (CNNs) and natural language processing via recurrent neural networks (RNNs). However, success stories of Deep Learning with standard feed-forward neural networks (FNNs) are rare. FNNs that perform well are typically shallow and, therefore cannot exploit many levels of abstract representations. We introduce self-normalizing neural networks (SNNs) to enable high-level abstract representations. While batch normalization requires explicit normalization, neuron activations of SNNs automatically converge towards zero mean and unit variance. The activation function of SNNs are "scaled exponential linear units" (SELUs), which induce self-normalizing properties. Using the Banach fixed-point theorem, we prove that activations close to zero mean and unit variance that are propagated through many network layers will converge towards zero mean and unit variance -- even under the presence of noise and perturbations. This convergence property of SNNs allows to (1) train deep networks with many layers, (2) employ strong regularization, and (3) to make learning highly robust. Furthermore, for activations not close to unit variance, we prove an upper and lower bound on the variance, thus, vanishing and exploding gradients are impossible. We compared SNNs on (a) 121 tasks from the UCI machine learning repository, on (b) drug discovery benchmarks, and on (c) astronomy tasks with standard FNNs and other machine learning methods such as random forests and support vector machines. SNNs significantly outperformed all competing FNN methods at 121 UCI tasks, outperformed all competing methods at the Tox21 dataset, and set a new record at an astronomy data set. The winning SNN architectures are often very deep. Implementations are available at: this http URL.

