1
$\begingroup$

I have a problem in my neural network relating to the bias vector.

I'm using this source as a reference.

My understanding of calculating the bias is that the partial derivative cost function with respect to the bias is equal to delta(l+1) (where l is the layer in question) from this equation; enter image description here

However the delta(l+1) matrix is of different dimensions to my bias matrix and since the bias is updated using the below equation this causes a problem. enter image description here

Now, I'm aware that the bias vector is essentially a n x 1 vector (where n is the number of neuron in the layer) but is treated as a n x m vector (where m is the number of training examples) by copying the values of the n x 1 into m rows. But when I calculate my delta(l)'s this gives me a n x m matrix that has different values in each location of the matrix. How can I then turn my bias matrix back into a n x 1 vector?

Example: A network with 1 inputs neuron, 2 hidden layers (with 3 neurons in each) and 1 output neuron will have a delta(2) that is a 3 x m matrix (where m is the number of training examples).

When since gradB(1) is equal to delta(1+1), gradB(1) will also be of size 3 x m. Each value in the matrix is different from the others so how do I update my bias vector b(1) (dimensions 3 x 1) from the 3 x m matrix? (ignoring all the scalar production relating to 1/m and alpha).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.