I am training a convolutional neural network on colored images and am trying to understand how and when the parameters of the neural network get updated. Suppose I have 1000 images. While creating the network, libraries like Keras take the dimensions of one image as the input layer dimensions (suppose 28 x 28 x 3) here. Does this mean that at a time only one image is passed through the network? The explanation here (http://cs231n.github.io/convolutional-networks/) also suggests that the CNN takes one image at a time. If this is indeed true, what happens to the network after this image is passed? Do parameters get updated after each image is passed?

As mentioned here (https://stackoverflow.com/questions/4752626/epoch-vs-iteration-when-training-neural-networks) the parameters of the neural network are updated after a batch of images is passed through it and not after every sample/image. How are multiple images even passed to a CNN? By how I mean in a schematic diagram, how would that look?

When we are talking about parameter update, does it mean only the weights or does that also imply learning rate, decay etc. ?

  • $\begingroup$ I work with tensorflow not keras, but all networks work with batches. For example for $32$ batch size your input becomes 4-th rank tensor $[32, 28, 28, 3]$ - you just add one more dimension. You have one update of weights, biases per batch (when backpropagating). Of course you can always set batch size to 1. $\endgroup$ – Łukasz Grad Apr 28 '17 at 8:17
  • $\begingroup$ Does this mean that if we use a filter size of (4 x 4), then each neuron in the convolutional layer will have a total of 32 x (4 x 4 x 3) weights associated with it? I think I am confused because of the inability to imagine the 4th dimension. $\endgroup$ – Arjun Mishra Apr 28 '17 at 10:41
  • $\begingroup$ No, all images in a batch are processed in parralel by the same $[4,4,3]$ kernel (for every output channel). Let's say u use valid kernel $[4,4,3,16]$ (16 output channels). Then the output of batch convolution is a $[32, 28, 28, 16]$ tensor, batch size doesn't change $\endgroup$ – Łukasz Grad Apr 28 '17 at 10:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.