I'm trying to convert my neural network implementation with ReLU to ELU.

I have visualisations of the ReLU decision bounaries and they look sensible. When I convert to ELU however, the bondaries look totally wrong.

The following are the only changes I'm making:

  1. alpha = 1

  2. In forward propagation, substitute:

    A1 = np.maximum(0, Z1) # ReLU
    A1 = np.where(Z1 > 0, Z1, alpha * (np.exp(Z1)-1)) # ELU
  3. In backward propagation, substitute:

    dgz = np.where(Z1 > 0, 1, 0) # ReLU
    dgz = np.where(Z1 > 0, 1, alpha * (np.exp(Z1)-1) + alpha) # ELU

What am I missing?

  • $\begingroup$ Could you post a pic of the boundaries. Are you sure the ELU approach converged? $\endgroup$ – Jim Mar 15 '18 at 16:19

The ICLR conference paper that you cite (Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs), 2016, Clevert et al.), provides the derivative in section 3:

$$ f'(x) = \begin{cases} 1 & \text{if } x > 0 \\ \alpha\exp(x) & \text{if } x \le 0. \\ \end{cases} $$

It appears that your implementation (simplified)

  dgz = np.where(Z1 > 0, 1, alpha*np.exp(Z1))

is correct.

However, other factors could explain the difference with $\text{ReLU}$. For example, it could be that the $\text{ELU}$-network requires more iterations to converge. Therefore, have a look at both learning curves (plotted by epoch or iteration).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.