I am using a simple CNN with one convolutional layer and one fully connected layer. I am using 3 filter channel and one input channel. I run my code 500 times with random initialization of weights in each loop. Each training loop is run for 30 thousand training steps. Then I plot filter weights of 1st filter and use fitting function to fit the data, similarly for second and third filter. I am getting similar result. Which shows that machine is learning same features to distinguish two classes. But I read every filter channel should learn a different feature. My model is showing 80 percent accuracy in 500 training loops. So what can be the reason for learning same features? Is my input data is quite simple?
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
5
Actually nothing guarantees that different convolutional filters would learn different features. The filters have same structure and are trained in the same way, so they are redundant before they learn something. We initialize the parameters randomly and we hope they will learn different features. You should probably try different initialization for the weights. If this doesn't help, try debugging the network as described in What should I do when my neural network doesn't learn?.
-
$\begingroup$ Actually i'm running a k -loop 500 times. in this k loop i use random initialization. Inside k-loop an i-loop is nested and every i-loop runs for 30 thousand training steps, I don''t use any random initialization in this i-loop. $\endgroup$– jerryMay 31, 2019 at 8:06
-
$\begingroup$ @jerry unfortunately I don't understand your comment. I meant randomly initializing the weights, if you didn't do that, then there is no chance in getting something else then exactly the same outcomes for all the filters. $\endgroup$– Tim ♦May 31, 2019 at 8:14
-
$\begingroup$ I want to say that I'm randomly initializing weights. $\endgroup$– jerryMay 31, 2019 at 20:02
-
$\begingroup$ Should I plot heat map to get a feeling of whether they are similar or not? $\endgroup$– jerryMay 31, 2019 at 20:03
-
$\begingroup$ @Jerry You can also plot a heat map of the "over all filters" std and mean to get a feel of the similarity. Only the std helps for that but I like to plot the mean too. $\endgroup$– dor00012Jul 13, 2022 at 23:27
$\begingroup$
$\endgroup$
1
Deep neural networks use Several hidden layers to hierarchically learn high level representation of a given picture, for example : first layer of your neural network might detect edges present in your given picture, second layer may detect curves present in your given picture and 3rd layer may detect your full face from your given picture,basically they try to bridge the gap between high level representation and low level features.so even if you keep on doing random initialization every time there is a possibility that first layer of your neural network might still detect something like edges present in your given picture over and over again,low level features are minor details of your picture, like lines,dots etc and these are typically caught by convolution filters... does it make sense?if you want your network to learn more features then increase neurons and hidden layers.
-
$\begingroup$ I'm only using one convolutional layer with 3 filter channels. So can I say that my network is learning almost same features(low level). My model's accuracy is quite higher even with such a simple network. $\endgroup$– jerryMay 31, 2019 at 20:00