Bias in matrix form of convolutional neural network

Question

With no bias nodes, the matrix form of the input data and the lowest-level parameters of following convolutional neural net

is the following:

$$ \left[\begin{array}{cccc} x_{11} & x_{12} & x_{14} & x_{14} \\ x_{21} & x_{22} & x_{24} & x_{24} \\ \vdots\\ x_{n1} & x_{n2} & x_{n4} & x_{n4} \\ \end{array}\right]\left[\begin{array}{cccc} w_1 & 0 & w_3 & 0\\ w_2 & 0 & w_4 & 0\\ 0 & w_1 & 0 & w_3\\ 0 & w_2 & 0 & w_4\\ \end{array}\right] $$

If I were to add bias terms, should it look like this:

$$ \left[\begin{array}{ccccc} 1 & x_{11} & x_{12} & x_{14} & x_{14} \\ 1 & x_{21} & x_{22} & x_{24} & x_{24} \\ &\vdots\\ 1 & x_{n1} & x_{n2} & x_{n4} & x_{n4} \\ \end{array}\right]\left[\begin{array}{cccc} b_1 &b_2 &b_3 &b_4 \\ w_1 & 0 & w_3 & 0\\ w_2 & 0 & w_4 & 0\\ 0 & w_1 & 0 & w_3\\ 0 & w_2 & 0 & w_4\\ \end{array}\right] $$

or this

$$ \left[\begin{array}{ccccc} 1 & x_{11} & x_{12} & x_{14} & x_{14} \\ 1 & x_{21} & x_{22} & x_{24} & x_{24} \\ &\vdots\\ 1 & x_{n1} & x_{n2} & x_{n4} & x_{n4} \\ \end{array}\right]\left[\begin{array}{cccc} b_1 &b_1 &b_2 &b_2 \\ w_1 & 0 & w_3 & 0\\ w_2 & 0 & w_4 & 0\\ 0 & w_1 & 0 & w_3\\ 0 & w_2 & 0 & w_4\\ \end{array}\right] $$

And why?

The first version has unique bias parameters for each time a linear function is applied to a region of the input data, while the second has a unique bias for each linear function.

Answer 1

The standard wisdom is to associate each neuron with its own bias term, in the sense that each neuron is responsible for computing its own feature over the input. Using the same bias term means you are applying a bias constraint to all such neurons - which may be beneficial to your problem or not. It does reduce the number of parameters, though.

I cannot imagine why it would be though, since any model with separate neuron biases could be fitted to match one with shared biases. So I would say that unless you have strong reasons to do so and limited training resources, use different biases per neuron.