Neural Network: Matlab uses different activation functions for different layers - why?

Question

I have trained on matlab an Artificial Neural Network with one input layer, one hidden layer and one output layer (my output is values between zero and one, which I turn into 0 or 1 according to a treshold of 0.5).

I have noticed that, by default, matlab used the 'tansig' transfer function for the hidden layer and then 'logsig' transfer function for the output layer. Can anyone give me an explanation for this?

Thank you in advance!

Answer 1

The big idea is that there's no particular requirement that all layers of a neural network use the same activation function. You can mix-and-match as you wish. That said, there are some reasons to prefer using $\tanh$ as the activation function of a hidden layer and $\sigma$ as the output function.

• The $\tanh(x)=\frac{\exp(x)-\exp(-x)}{\exp(x)+\exp(-x)}$ function is a standard activation function. Using it in a neural network is no more surprising than using least squares as an objective function for a regression task.

• The function $\sigma(x)=\frac{1}{1+\exp(-x)}$ is a standard way to map real numbers to real values in (0,1). So it's commonly used to model probabilities. Since your task is to predict 0 or 1, using this model suggests modeling the probability that the sample is labeled 1.

• Using a $\tanh$ function in the last layer would be implausible, because it does not have a clear relationship to modeling the probability that a sample is labeled 1. The function $\tanh$ returns values between -1 and 1, so it is not a probability.

• If you wished, you could use $\sigma(x)$ as an activation function. But $\tanh$ is preferred because having a stronger gradient and giving positive and negative outputs makes it easier to optimize. See: tanh activation function vs sigmoid activation function

• But also note that ReLU and similar functions are generally preferred as activation functions in hidden layers. See: What are the advantages of ReLU over sigmoid function in deep neural networks?

• The choice to use $\tanh$ as a default is likely more about software development practices than mathematical principles: changing the default behavior of software can break legacy code and cause unexpected behavior. ReLU units only became popular recently, relative to the age of MATLAB. The Neural Network Toolbox add-on first published 1992 (since then, it's been rebranded as the "Deep Learning Toolbox"). In 1992, building a neural network was almost synonymous with a single-layer network with $\tanh$ or $\sigma$ activation functions.

  But there's unlikely to be any definitive explanation for why MATLAB chose this default unless they happened to publish a justification for this choice (e.g. release notes or documentation).