I'm a newbie in deep learning. As I have known, each neuron has a gain/weight and an offset/bias with an activation function (e.g. sigmoid, tanh, ReLU, identity and etc).

In the convolution layer in a Convolution-Neural-Networks mentioned that it usually concatenate with ReLU activation function, but what happened in the fully connected layer?

What is the most common activation function in a fully connected layer in a deep CNN?

  • Fully connected input layer (flatten)━takes the output of the previous layers, “flattens” them and turns them into a single vector that can be an input for the next stage.
  • The first fully connected layer━takes the inputs from the feature analysis and applies weights to predict the correct label.
  • Fully connected output layer━gives the final probabilities for each label.
  • 1
    $\begingroup$ Most people would call y=x the "identity" activation. $\endgroup$ – Sycorax Aug 27 '19 at 12:58
  • $\begingroup$ That's right, I'll update it. $\endgroup$ – Benyamin Jafari Aug 27 '19 at 13:43

You can use whatever activation function you want in any layer.

ReLUs and similar functions are popular because they speed up network training, as they only have a flat gradient on one side instead of two, as is the case with $\tanh$ and sigmoid activations.

Using identity activations results in a linear model, because linear functions are closed under composition.

  • $\begingroup$ Thanks for response. $\endgroup$ – Benyamin Jafari Aug 27 '19 at 13:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.