1
$\begingroup$

With no bias nodes, the matrix form of the input data and the lowest-level parameters of following convolutional neural net

enter image description here

is the following:

$$ \left[\begin{array}{cccc} x_{11} & x_{12} & x_{14} & x_{14} \\ x_{21} & x_{22} & x_{24} & x_{24} \\ \vdots\\ x_{n1} & x_{n2} & x_{n4} & x_{n4} \\ \end{array}\right]\left[\begin{array}{cccc} w_1 & 0 & w_3 & 0\\ w_2 & 0 & w_4 & 0\\ 0 & w_1 & 0 & w_3\\ 0 & w_2 & 0 & w_4\\ \end{array}\right] $$

If I were to add bias terms, should it look like this:

$$ \left[\begin{array}{ccccc} 1 & x_{11} & x_{12} & x_{14} & x_{14} \\ 1 & x_{21} & x_{22} & x_{24} & x_{24} \\ &\vdots\\ 1 & x_{n1} & x_{n2} & x_{n4} & x_{n4} \\ \end{array}\right]\left[\begin{array}{cccc} b_1 &b_2 &b_3 &b_4 \\ w_1 & 0 & w_3 & 0\\ w_2 & 0 & w_4 & 0\\ 0 & w_1 & 0 & w_3\\ 0 & w_2 & 0 & w_4\\ \end{array}\right] $$

or this

$$ \left[\begin{array}{ccccc} 1 & x_{11} & x_{12} & x_{14} & x_{14} \\ 1 & x_{21} & x_{22} & x_{24} & x_{24} \\ &\vdots\\ 1 & x_{n1} & x_{n2} & x_{n4} & x_{n4} \\ \end{array}\right]\left[\begin{array}{cccc} b_1 &b_1 &b_2 &b_2 \\ w_1 & 0 & w_3 & 0\\ w_2 & 0 & w_4 & 0\\ 0 & w_1 & 0 & w_3\\ 0 & w_2 & 0 & w_4\\ \end{array}\right] $$

And why?

The first version has unique bias parameters for each time a linear function is applied to a region of the input data, while the second has a unique bias for each linear function.

$\endgroup$

1 Answer 1

0
$\begingroup$

The standard wisdom is to associate each neuron with its own bias term, in the sense that each neuron is responsible for computing its own feature over the input. Using the same bias term means you are applying a bias constraint to all such neurons - which may be beneficial to your problem or not. It does reduce the number of parameters, though.

I cannot imagine why it would be though, since any model with separate neuron biases could be fitted to match one with shared biases. So I would say that unless you have strong reasons to do so and limited training resources, use different biases per neuron.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.