Timeline for How do the derivatives of the loss function with respect to a layer's inputs form a Jacobian?
Current License: CC BY-SA 4.0
9 events
when toggle format | what | by | license | comment | |
---|---|---|---|---|---|
Jan 25, 2021 at 12:51 | vote | accept | AlwaysLearning | ||
Jan 24, 2021 at 19:42 | comment | added | AlwaysLearning | I don't understand. The weight matrices stay two-dimensional. It is the delta by which they are updated that depends on all the training samples, but that delta is also a matrix... My question is how to denote the matrix of partial derivatives of the loss with respect to all the inputs, since it is not a Jacobian. | |
Jan 24, 2021 at 17:23 | comment | added | gunes | Ok, my advice would be concatenating [J1,J2...] where J_i is the loss wrt sample i, and vectorise your weight matrices (then reshape while updating) so that you don't deal with 3d tensors. The implementation will be quite tricky. You can post it as another question (with your simple example code) after giving some thought maybe. | |
Jan 24, 2021 at 16:53 | comment | added | AlwaysLearning | I would like to derive matrix equations for batch updates of the weights. I would like to do so in matrix form from the beginning to the end, without going first through the derivations for a single training example. This is where the mentioned matrix comes into play. | |
Jan 24, 2021 at 15:39 | comment | added | gunes | You don't need to construct such a matrix because gradients for different training examples are averaged (or summed) in backpropagation. | |
Jan 24, 2021 at 15:37 | comment | added | AlwaysLearning | You are correct. When perusing that tutorial (among many others) I missed that he is talking about stochastic updates... In any case, what notation would you use for the matrix I have in mind? | |
Jan 24, 2021 at 14:10 | comment | added | gunes | @AlwaysLearning can you point out where the author of the tutorial you linked thinks the Jacobian has different entries for each training example? | |
Jan 24, 2021 at 8:39 | comment | added | AlwaysLearning | My question was about joining the derivatives for all the training samples into one matrix form. Then we get a matrix, not just a row vector. How is that matrix a Jacobian? (I have edited the question to clarify) | |
Jan 23, 2021 at 22:42 | history | answered | gunes | CC BY-SA 4.0 |