Skip to main content
9 events
when toggle format what by license comment
Jan 25, 2021 at 12:51 vote accept AlwaysLearning
Jan 24, 2021 at 19:42 comment added AlwaysLearning I don't understand. The weight matrices stay two-dimensional. It is the delta by which they are updated that depends on all the training samples, but that delta is also a matrix... My question is how to denote the matrix of partial derivatives of the loss with respect to all the inputs, since it is not a Jacobian.
Jan 24, 2021 at 17:23 comment added gunes Ok, my advice would be concatenating [J1,J2...] where J_i is the loss wrt sample i, and vectorise your weight matrices (then reshape while updating) so that you don't deal with 3d tensors. The implementation will be quite tricky. You can post it as another question (with your simple example code) after giving some thought maybe.
Jan 24, 2021 at 16:53 comment added AlwaysLearning I would like to derive matrix equations for batch updates of the weights. I would like to do so in matrix form from the beginning to the end, without going first through the derivations for a single training example. This is where the mentioned matrix comes into play.
Jan 24, 2021 at 15:39 comment added gunes You don't need to construct such a matrix because gradients for different training examples are averaged (or summed) in backpropagation.
Jan 24, 2021 at 15:37 comment added AlwaysLearning You are correct. When perusing that tutorial (among many others) I missed that he is talking about stochastic updates... In any case, what notation would you use for the matrix I have in mind?
Jan 24, 2021 at 14:10 comment added gunes @AlwaysLearning can you point out where the author of the tutorial you linked thinks the Jacobian has different entries for each training example?
Jan 24, 2021 at 8:39 comment added AlwaysLearning My question was about joining the derivatives for all the training samples into one matrix form. Then we get a matrix, not just a row vector. How is that matrix a Jacobian? (I have edited the question to clarify)
Jan 23, 2021 at 22:42 history answered gunes CC BY-SA 4.0