In normal neural network, in the first layer we have data, random weights and bias term. This then pass through hidden layer, then out output layer, we then calculate the error, and finally based on error we again update the weights.

This looping is done continuously till we get satisfactory error rate.

But in CNN, we first use different filters (say 5 filters) to extract features. After 2 convulation layer and max-pooling plus padding plus flattening, we get a vector. This vector is input vector as we observe in normal NN. We also have random weights and bias terms. This value is then propagated into fully connected layer and follows the same pattern as we observe in normal NN.

My questions are the following:

1) In case of normal NN, the weights and biases are updated during backward propagation. In CNN, apart from weights and biases does filters values are also updated?

2) Related to first question, how the filters are chosen? "Filters are chosen" means types of filters are chosen. As per my understanding different filters extract different features of an image. As well as how number of filters are chosen.

  • $\begingroup$ The filters are nothing more than weights, so yes, they do get updated. Filters are chosen only in the sense of specifying how many filters you want at each level of the CNN. Backprop will determine the weights and hence the filters. $\endgroup$ – Alex R. May 13 '17 at 19:30
  • $\begingroup$ @AlexR.: THanks a lot for the answer! So you are saying the we initially taking a standard "filter-weights". These filter-weights do standard feature mapping. The it goes through different hidden layer to final classification and realised that the initial filter-weights were not effective. And so they are also updated through backprop. Also, can you please put the comment in answer part, so that I can accept it. For the completeness of the question. $\endgroup$ – Beta May 13 '17 at 19:38

As @Alex R. already explained in his comment, convolution filters are also weights that get updated during the training. There are several ways how the filters are chosen, and there are two aspects to consider: architecture and weight initialization.

Architecture of the convolution filters means how many filters does the network have, how large they should be, how many layers of them. This is mostly driven by current rules of thumb, as I described in this answer.

Initialization of the convolution filters means how should they look at the start of the training. There are two main streams:

  1. Random initializations (notably "xavier" initialization from Glorot and Bengio 2010 paper, explained nicely in this blog; and "msra" from the ResNet paper by He et al., 2015.)
  2. Transfer learning, domain adaptation (using convolution filters from a network trained on some other task in the same input domain). A list of sources of pre-trained networks can be found in this answer.

A more thorough overview of possible intialization schemes from the perspective of their regularization abilities is in my article Regularization for Deep Learning: A Taxonomy, Section 7.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.