Skip to main content
Cross Validated

Return to Question

added 77 characters in body
Source Link
Sebastian Nielsen
Sebastian Nielsen
  • 111
  • 6

This is my network represented in matrices: (a dot represents an arbitrary number) enter image description here Feed-forwarding: (I omitted nesting it all in an activation function for the sake of brevity) enter image description here Backpropagation enter image description here

The question

enter image description here $\partial E/\partial b^3$ should equal a matrix of dimensions: $3 \times 1$, in order to make the subtraction possible by the former $b^3$. The batch size of 5, however, made the dimensions of the $b^3$ matrix equal to $3 \times 5$, which is problematic as you now can't subtract $\partial E/\partial b^3$ because of the dimension mismatch.

What went wrong? Am I supposed to take the average of each row of the $\partial E/\partial b^3$ matrix or perhaps just accumulate each row? Or is it something completely different?

My reflection At the moment I am thinking that what I have done up to now is correct, however I just need to either accumulate or take the average of all the numbers in each row in the matrix $∂E/∂b3$. This way the matrix will be of size $3\times1$ as wanted, and it also makes intuitive sense in the way that I am updating the bias based on an error calculated over an entire batch size, therefor either accumulating or taking the average would make sense. However as I mentioned I am not sure which one it is, or if it is even the right choice.

Any help is highly appreciated!

This is my network represented in matrices: (a dot represents an arbitrary number) enter image description here Feed-forwarding: enter image description here Backpropagation enter image description here

The question

enter image description here $\partial E/\partial b^3$ should equal a matrix of dimensions: $3 \times 1$, in order to make the subtraction possible by the former $b^3$. The batch size of 5, however, made the dimensions of the $b^3$ matrix equal to $3 \times 5$, which is problematic as you now can't subtract $\partial E/\partial b^3$ because of the dimension mismatch.

What went wrong? Am I supposed to take the average of each row of the $\partial E/\partial b^3$ matrix or perhaps just accumulate each row? Or is it something completely different?

My reflection At the moment I am thinking that what I have done up to now is correct, however I just need to either accumulate or take the average of all the numbers in each row in the matrix $∂E/∂b3$. This way the matrix will be of size $3\times1$ as wanted, and it also makes intuitive sense in the way that I am updating the bias based on an error calculated over an entire batch size, therefor either accumulating or taking the average would make sense. However as I mentioned I am not sure which one it is, or if it is even the right choice.

Any help is highly appreciated!

This is my network represented in matrices: (a dot represents an arbitrary number) enter image description here Feed-forwarding: (I omitted nesting it all in an activation function for the sake of brevity) enter image description here Backpropagation enter image description here

The question

enter image description here $\partial E/\partial b^3$ should equal a matrix of dimensions: $3 \times 1$, in order to make the subtraction possible by the former $b^3$. The batch size of 5, however, made the dimensions of the $b^3$ matrix equal to $3 \times 5$, which is problematic as you now can't subtract $\partial E/\partial b^3$ because of the dimension mismatch.

What went wrong? Am I supposed to take the average of each row of the $\partial E/\partial b^3$ matrix or perhaps just accumulate each row? Or is it something completely different?

My reflection At the moment I am thinking that what I have done up to now is correct, however I just need to either accumulate or take the average of all the numbers in each row in the matrix $∂E/∂b3$. This way the matrix will be of size $3\times1$ as wanted, and it also makes intuitive sense in the way that I am updating the bias based on an error calculated over an entire batch size, therefor either accumulating or taking the average would make sense. However as I mentioned I am not sure which one it is, or if it is even the right choice.

Any help is highly appreciated!

deleted 6 characters in body
Source Link
Sebastian Nielsen
Sebastian Nielsen
  • 111
  • 6

This is my network represented in matrices: (a dot represents an arbitrary number) enter image description here

enter image description here Feed-forwarding: enter image description hereenter image description here Backpropagation enter image description here

The question

enter image description here $\partial E/\partial b^3$ should equal a matrix of dimensions: $3 \times 1$, in order to make the subtraction possible by the former $b^3$. The batch size of 5, however, made the dimensions of the $b^3$ matrix equal to $3 \times 5$, which is problematic as you now can't subtract $\partial E/\partial b^3$ because of the dimension mismatch.

What went wrong? Am I supposed to take the average of each row of the $\partial E/\partial b^3$ matrix or perhaps just accumulate each row? Or is it something completely different?

My reflection At the moment I am thinking that what I have done up to now is correct, however I just need to either accumulate or take the average of all the numbers in each row in the matrix $∂E/∂b3$. This way the matrix will be of size $3\times1$ as wanted, and it also makes intuitive sense in the way that I am updating the bias based on an error calculated over an entire batch size, therefor either accumulating or taking the average would make sense. However as I mentioned I am not sure which one it is, or if it is even the right choice.

Any help is highly appreciated!

This is my network represented in matrices: (a dot represents an arbitrary number) enter image description here

Feed-forwarding: enter image description here Backpropagation enter image description here

The question

enter image description here $\partial E/\partial b^3$ should equal a matrix of dimensions: $3 \times 1$, in order to make the subtraction possible by the former $b^3$. The batch size of 5, however, made the dimensions of the $b^3$ matrix equal to $3 \times 5$, which is problematic as you now can't subtract $\partial E/\partial b^3$ because of the dimension mismatch.

What went wrong? Am I supposed to take the average of each row of the $\partial E/\partial b^3$ matrix or perhaps just accumulate each row? Or is it something completely different?

My reflection At the moment I am thinking that what I have done up to now is correct, however I just need to either accumulate or take the average of all the numbers in each row in the matrix $∂E/∂b3$. This way the matrix will be of size $3\times1$ as wanted, and it also makes intuitive sense in the way that I am updating the bias based on an error calculated over an entire batch size, therefor either accumulating or taking the average would make sense. However as I mentioned I am not which one it is, or if it is even the right choice.

Any help is highly appreciated!

This is my network represented in matrices: (a dot represents an arbitrary number) enter image description here Feed-forwarding: enter image description here Backpropagation enter image description here

The question

enter image description here $\partial E/\partial b^3$ should equal a matrix of dimensions: $3 \times 1$, in order to make the subtraction possible by the former $b^3$. The batch size of 5, however, made the dimensions of the $b^3$ matrix equal to $3 \times 5$, which is problematic as you now can't subtract $\partial E/\partial b^3$ because of the dimension mismatch.

What went wrong? Am I supposed to take the average of each row of the $\partial E/\partial b^3$ matrix or perhaps just accumulate each row? Or is it something completely different?

My reflection At the moment I am thinking that what I have done up to now is correct, however I just need to either accumulate or take the average of all the numbers in each row in the matrix $∂E/∂b3$. This way the matrix will be of size $3\times1$ as wanted, and it also makes intuitive sense in the way that I am updating the bias based on an error calculated over an entire batch size, therefor either accumulating or taking the average would make sense. However as I mentioned I am not sure which one it is, or if it is even the right choice.

Any help is highly appreciated!

deleted 150 characters in body
Source Link
Sebastian Nielsen
Sebastian Nielsen
  • 111
  • 6

This is my network represented in matrices: (a dot represents an arbitrary number) enter image description here

Feed-forwarding: enter image description here Backpropagation enter image description here

The question

enter image description here

What went wrong? $(\partial E)/(\partial b^3)$ $\partial E/\partial b^3$ should equal a matrix of dimensions: $3 \times 1$, in order to make the subtraction possible by the former b^3$b^3$. The batch size of 5, however, made the dimensions of the $b^3$ matrix equal to $3 \times 5$, which is problematic as you now can't subtract $\partial E/\partial b^3$ because of the dimension mismatch.

What is the proper waywent wrong? Am I supposed to update a biastake the average of each row of the $\partial E/\partial b^3$ matrix or perhaps just accumulate each row? Or is it something completely different?

My reflection At the moment I am thinking that what I have done up to now is correct, however I just need to either accumulate or take the average of all the numbers in each row in the matrix $∂E/∂b3$. This way the matrix will be of size $3\times1$ as wanted, and it also makes intuitive sense in the way that I am updating the bias based on an error calculated over an entire batch size, therefor either accumulating or taking the average would make sense. However as I mentioned I am not which one it is, or if it is even the right choice.

Any help is highly appreciated!

This is my network represented in matrices: (a dot represents an arbitrary number) enter image description here

Feed-forwarding: enter image description here Backpropagation enter image description here

The question

enter image description here

What went wrong? $(\partial E)/(\partial b^3)$ should equal a matrix of dimensions: $3 \times 1$, in order to make the subtraction possible by the former b^3. The batch size of 5, however, made the dimensions of the $b^3$ matrix equal to $3 \times 5$, which is problematic as you now can't subtract $\partial E/\partial b^3$ because of the dimension mismatch.

What is the proper way to update a bias?

This is my network represented in matrices: (a dot represents an arbitrary number) enter image description here

Feed-forwarding: enter image description here Backpropagation enter image description here

The question

enter image description here $\partial E/\partial b^3$ should equal a matrix of dimensions: $3 \times 1$, in order to make the subtraction possible by the former $b^3$. The batch size of 5, however, made the dimensions of the $b^3$ matrix equal to $3 \times 5$, which is problematic as you now can't subtract $\partial E/\partial b^3$ because of the dimension mismatch.

What went wrong? Am I supposed to take the average of each row of the $\partial E/\partial b^3$ matrix or perhaps just accumulate each row? Or is it something completely different?

My reflection At the moment I am thinking that what I have done up to now is correct, however I just need to either accumulate or take the average of all the numbers in each row in the matrix $∂E/∂b3$. This way the matrix will be of size $3\times1$ as wanted, and it also makes intuitive sense in the way that I am updating the bias based on an error calculated over an entire batch size, therefor either accumulating or taking the average would make sense. However as I mentioned I am not which one it is, or if it is even the right choice.

Any help is highly appreciated!

deleted 150 characters in body
Source Link
Sebastian Nielsen
Sebastian Nielsen
  • 111
  • 6
added 146 characters in body
Source Link
Sebastian Nielsen
Sebastian Nielsen
  • 111
  • 6
added 146 characters in body
Source Link
Sebastian Nielsen
Sebastian Nielsen
  • 111
  • 6
deleted 2 characters in body
Source Link
Sebastian Nielsen
Sebastian Nielsen
  • 111
  • 6
deleted 2 characters in body
Source Link
Sebastian Nielsen
Sebastian Nielsen
  • 111
  • 6
Source Link
Sebastian Nielsen
Sebastian Nielsen
  • 111
  • 6