0
$\begingroup$

Processing each mini-batch gives us the best weights/biases result for the input used in that mini-batch. How to reconcile the results obtained for all mini-batches? Do you take the average to come up with the final weights/biases for the trained network?

$\endgroup$
7
  • $\begingroup$ You'll have to be a little more specific about what you're trying to do with minibatches. Are you asking about minibatch training (e.g. minibatch gradient descent) or something else? $\endgroup$
    – user20160
    Commented Jan 2, 2018 at 8:48
  • $\begingroup$ Let me clarify my question. Let's say we have two mini-batches. I processed the first mini-batch and obtained the update to the weights/aliases used in that mini-batch. Now I have two choices: $\endgroup$
    – i262666
    Commented Jan 3, 2018 at 19:03
  • $\begingroup$ 1) Process the second mini-batch completely independently (meaning start processing the second mini-batch initially using again randomly selected weights/biases parameters. Then obtain updates to the weights/biases for the second mini-batch and apply them to the weights/biases in the second mini-batch. Next, take the average of weights/biases obtained in the first and second mini-batches and consider that average as the final weights/biases for the trained network. See the last part of my clarification below. $\endgroup$
    – i262666
    Commented Jan 3, 2018 at 19:37
  • $\begingroup$ 2) After the first mini-batch has been processed and the adjusted weights/biases for the first mini-batch have been calculated, use these weights/biases as the initial input for the second mini-batch. Consider the adjusted weights/biases obtained for the second mini-batch as the final weights/biases for the trained network. $\endgroup$
    – i262666
    Commented Jan 3, 2018 at 19:38
  • $\begingroup$ I am confused about the "as the final weights/biases for the trained network" part. The second approach is the typical approach, but then over hundreds of minibatches and iterations. I have never heard of anybody using the first approach - my intuition says that the average of the weights of two networks is meaningless when initialized in different random ways. $\endgroup$
    – GR4
    Commented Jan 4, 2018 at 9:10

1 Answer 1

0
$\begingroup$

Essentially, each mini-batch contains the average of the gradients of the individual errors. Therefore if you had two mini-batches, you could take the average of the gradient updates of both mini-batches to tweak the weights to reduce the error for those samples.

Note however that each back-propagation step tweaks the weights in the right direction to diminish the error rather than computing the absolute best weights/biases setting for this particular minibatch. Of course you could repeat the process to optimize the weights for that minibatch but that is not what you want as you would obtain a very biased result for the samples in the minibatch.

$\endgroup$
1
  • $\begingroup$ Thank you very much for you response. I put some clarifications to my question above. Would appreciate if you can answer it. $\endgroup$
    – i262666
    Commented Jan 3, 2018 at 19:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.