From the Book"fundamentals of neural network", the input layer of a feedforward neural network has linear activation function. Elman recurrent NN is the same as a feedforward except that it has context layer. What should be the activation functions of each layer in an elman recurrent NN?

I have searched for this answer from many sources but many journal papers do not mention about the activation functions. Some information online says that the hidden layers should have tansig activation function.

So, What should be the activation functions of each layer in an elman recurrent NN?


An Elman network leaves the choice of the activation function to the user, since it only specifies these equations for the recurrence:

\begin{align} h_t &= \sigma_h(W_{h} x_t + U_{h} h_{t-1} + b_h) \\ y_t &= \sigma_y(W_{y} h_t + b_y) \end{align}

Variables and functions:

  • $x_t$: input vector
  • $h_t$: hidden layer vector
  • $y_t$: output vector
  • $W$, $U$ and $b$: parameter matrices and vector
  • $\sigma_h$ and $\sigma_y$: Activation functions

{1}, which is one of the most cited references defining an Elman network, doesn't seem to indicate that an Elman network should use some specific activation functions either.

FYI Comprehensive list of activation functions in neural networks with pros/cons


  • $\begingroup$ Yes , i agree and thanks for replying. I saw these equations in most journal papers. But I needed some proof or some reliable sources to confirm your answer :/ $\endgroup$ – user7085565 Dec 21 '16 at 19:30
  • $\begingroup$ @user7085565: sounds good I added one reference $\endgroup$ – Franck Dernoncourt Dec 21 '16 at 19:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.