Understanding how many biases are there in a neural network?

Question

I am trying to understand biases in neural nets, but different websites show very different answers.

For example, how many biases is there in a fully connected neural network with a single input layer with 5 units and a single output layer with 4 units? And what about a fully connected neural network with a single input layer with 5 units, a single hidden layer with 4 units, and a single output layer with 3 units?

For example, if I understand this correctly, https://ai.stackexchange.com/questions/17584/why-does-the-bias-need-to-be-a-vector-in-a-neural-network, the answer of the first should be 5 and for the second 4 + 3. Each neuron except for in the input-layer has a bias.

However, at https://ayearofai.com/rohan-5-what-are-bias-units-828d942b4f52, it is explained such that each layer including the input-layer has one bias. So the answer to the example above is one in the first and two in the second.

What is correct? What am I misunderstanding here?

Answer 1

Each node in the hidden layers or in the output layer of a feed-forward neural network has its own bias term. (The input layer has no parameters whatsoever.) At least, that's how it works in TensorFlow. To be sure, I constructed your two neural networks in TensorFlow as follows:

model1 = tf.keras.models.Sequential([
        tf.keras.layers.Dense(4, activation = 'softmax', input_shape = (5,))])

model2 =  tf.keras.models.Sequential([
        tf.keras.layers.Dense(4, activation = 'relu', input_shape = (5,)),
        tf.keras.layers.Dense(3,activation = 'softmax')])

Here is the summary of these two models that TensorFlow provides:

The first model has 24 parameters, because each node in the output layer has 5 weights and a bias term (so each node has 6 parameters), and there are 4 nodes in the output layer. The second model has 24 parameters in the hidden layer (counted the same way as above) and 15 parameters in the output layer. Each node in the output layer has 4 weights and a bias term (so 5 parameters per node in the output layer), and there are 3 nodes in the output layer.

Answer 2

It depends on whether you're counting units or learnable parameters. In the first example/link, we are counting the parameters, and in the second example we are counting the units.

There's one bias unit per layer. Including the input ensures that a network with one layer has a bias.