Common activation function in fully connected layer

Question

I'm a newbie in deep learning. As I have known, each neuron has a gain/weight and an offset/bias with an activation function (e.g. sigmoid, tanh, ReLU, identity and etc).

In the convolution layer in a Convolution-Neural-Networks mentioned that it usually concatenate with ReLU activation function, but what happened in the fully connected layer?

What is the most common activation function in a fully connected layer in a deep CNN?

• Fully connected input layer (flatten)━takes the output of the previous layers, “flattens” them and turns them into a single vector that can be an input for the next stage.
• The first fully connected layer━takes the inputs from the feature analysis and applies weights to predict the correct label.
• Fully connected output layer━gives the final probabilities for each label.

Answer 1

You can use whatever activation function you want in any layer.

ReLUs and similar functions are popular because they speed up network training, as they only have a flat gradient on one side instead of two, as is the case with $\tanh$ and sigmoid activations.

Using identity activations results in a linear model, because linear functions are closed under composition.

Answer 2

First note that a fully connected neural network usually has more than one activation functions (the activation function in hidden layers is often different from that used in the output layer).

Any function that is continuous can be used as an activation function, including linear function g(z)=z, which is often used in an output layer.

Activation functions in hidden layers are usually nonlinear, e.g. relu, which is a piecewise linear function that introduces the most simple nonlinearity.

Other often used activation functions include sigmoid, tanh, softmax, etc.