Neural networks - what is the point of having sigmoid activation function $\sigma(.)$ AND sigmoid g(t)?

Question

Just so we're all on the same page, this is the classic neural network set up as I understand it:

$$z_j=\sigma(\alpha_0j+\alpha_j^T x)$$ $$t=\beta_0+\Sigma^k_{j=1}\beta_j z_j)=\beta_0+\beta^Tx $$ $$ y=f(x)=g(t) $$

(I may have missed a few bits off like the hat on the y and some bold to show the vectors, but you get the idea)

$\sigma(.)$ is the activation function which can be a sigmoid function (can also be tanh, ReLu or threshold), but then in the final line there is another sigmoid function in g(t).

What is the logic in having two sigmoid functions? I get that this is what happens, but haven't yet seen any justification for the setup like this.

Please dumb it down as much as possible for me. I do not understand the meaning of the word "patronise".

Sometimes it feels like neural networks are just dumping input into a massive bag of arbitrary linear algebra, stirring it around, and hoping the output is something that you expect it to be...

Answer 1

The sigmoid functions in the hidden layers introduce nonlinearity. That is, they bend the output and let output values increase and then decrease and then increase again (or whatever), behavior that is not possible with a linear relationship.

The sigmoid function on the output neuron compresses the final value into the interval $(0,1)$. This is often (but not necessarily) to give an output that is a probability in a so-called "classification" problem that has a binary output category (e.g., dog or cat) and gives the probability of being in either category. If you know or someday learn about generalized linear models, this is related to the link function. I once posted about this on the AI Stack.

That the function in the hidden neurons and the function in the output neuron can be the same is more coincidence than anything, and they do not have to be the same.

Answer 2

The outline you've described sounds like a neural network with 1 hidden layer with an activation function $\sigma$ and 1 standard logistic function output neuron $g$ : $$ \hat y = g(W_2 \sigma(W_1x+b_1)+b_2). $$

where $g(z) = \frac{1}{1+\exp(-z)} = \frac{\exp(z)}{1 + \exp(z)}$.

Some losses require that the model outputs a probability, such as the binary cross-entropy loss. Using the logistic function $g$ enforces that the prediction $\hat y$ is a probability, the sigmoid function gives real numbers between 0 and 1 (and sum to 1 across mutually exclusive outcomes), i.e. probabilities. The logistic function is not the only continuous, monotonic function that maps real numbers to numbers between 0 and 1 (and sum to 1 across mutually exclusive outcomes), it's just a very commonly used function. For some other common examples, see Difference between logit and probit models.

If the loss we're using does not require that the predictions are probabilities, then we don't have to use $g$ as the output function. An alternative function, including the identity function, could be used.

The hidden layers use a nonlinear activation function $\sigma$ to allow the network to learn a nonlinear basis. See: What does the hidden layer in a neural network compute? If we use a linear activation function, such as the identity function, then the neural network is restricted to being a composition of linear operations, which is itself linear.

There's no requirement that $g$ and $\sigma$ are the same function. Modern neural network researchers will choose $g$ and $\sigma$ according to whatever they need the network to do, and it's more typical that $g$ and $\sigma$ will not have any requirement to be the same function.