is Sigmoid activation function better than Leaky Relu?

Question

I recently implemented a neural network from scratch in python with one hidden layer on the iris dataset for classification. Initially i applied Leaky relu for hidden layer acrivations and got an accuracy around 40% then i switched to sigmoid and found that accuracy improved drastically to 93% accuracy.

How is that possible?

I learned from Andrew ng lectures that Leaky relu is a best option.

Answer 1

My hypothesis is that you found a configuration (learning rate, batch size, number of hidden nodes, etc.) which happened to be better for the sigmoid network than the Leaky ReLU network. I assume that there's an alternative configuration for which the Leaky ReLU network is better than the sigmoid network.

As an aside, the main motivation of ReLU-type activations is that they work better in deep networks, where sigmoid and tanh networks tend to get saturated and the gradient vanishes. Using a network with 1 hidden layer is not necessarily going to highlight the contrast between sigmoid and ReLU activations.

I would caution against drawing any general conclusions from a single experiment using the Iris data. It's a small data set where one of the classes is linearly separable from the rest, so it's only useful as a toy problem; it's just not very complex.

Answer 2

There is no "best" activation function. If there were, all the neural network architectures would stick to the single "best" one, while what we see is something opposite: different neural networks, or even different layers of single network, use different activation functions. Some of them, like ReLU, are more popular then others, but there are cases where they don't work and you need to use different activation functions instead. People usually do what you did: start with what is currently known to be most promising solution, but if it fails, try also the alternatives.