# Error in converting activation function from ReLU to ELU

I'm trying to convert my neural network implementation with ReLU to ELU.

I have visualisations of the ReLU decision bounaries and they look sensible. When I convert to ELU however, the bondaries look totally wrong.

The following are the only changes I'm making:

1. alpha = 1

2. In forward propagation, substitute:

A1 = np.maximum(0, Z1) # ReLU
A1 = np.where(Z1 > 0, Z1, alpha * (np.exp(Z1)-1)) # ELU

3. In backward propagation, substitute:

dgz = np.where(Z1 > 0, 1, 0) # ReLU
dgz = np.where(Z1 > 0, 1, alpha * (np.exp(Z1)-1) + alpha) # ELU


What am I missing?

• Could you post a pic of the boundaries. Are you sure the ELU approach converged? – Jim Mar 15 '18 at 16:19

$$f'(x) = \begin{cases} 1 & \text{if } x > 0 \\ \alpha\exp(x) & \text{if } x \le 0. \\ \end{cases}$$
  dgz = np.where(Z1 > 0, 1, alpha*np.exp(Z1))

However, other factors could explain the difference with $\text{ReLU}$. For example, it could be that the $\text{ELU}$-network requires more iterations to converge. Therefore, have a look at both learning curves (plotted by epoch or iteration).