I'm trying to convert my neural network implementation with ReLU to ELU.
I have visualisations of the ReLU decision bounaries and they look sensible. When I convert to ELU however, the bondaries look totally wrong.
The following are the only changes I'm making:
alpha = 1
In forward propagation, substitute:
A1 = np.maximum(0, Z1) # ReLU A1 = np.where(Z1 > 0, Z1, alpha * (np.exp(Z1)-1)) # ELU
In backward propagation, substitute:
dgz = np.where(Z1 > 0, 1, 0) # ReLU dgz = np.where(Z1 > 0, 1, alpha * (np.exp(Z1)-1) + alpha) # ELU
What am I missing?