5
$\begingroup$

I'm following Google's TensorFlow Deep MNIST for Experts tutorial.

Here is my code:

http://pastebin.com/ePktssrn

The networks seems to get close to 100% accuracy after about training 1000 steps, but all of its kernels on the outermost layer remain completely unchanged.

Here's a visualization I made: http://i.imgur.com/gCmxq40.png Each row represents one of 32 5x5 kernels defined by W_conv1; leftmost image in the row is initial state of the kernel, and each next image horizontally is how that kernel looks after 5 training steps.

As it can be clearly seen from the picture, kernels seem to be completely static.

I used the same code to draw kernels for a shallow network with convolutions, and they train very well - you can see shapes slowly forming out of noise. But on that particular example, kernels stay completely same.

Is this normal? Is this a flaw in google's example of deep CNN? Is this some mistake I made following the tutorial?

$\endgroup$
  • 1
    $\begingroup$ Please paste in whatever context is necessary to understand & answer your question. We want this thread to remain valuable even if the link goes dead. $\endgroup$ – gung Jul 1 '16 at 12:44
  • $\begingroup$ I'm pretty sure I posted it. Images and link to code are all optional to understand what I mean: Google's Deep MNIST for Experts model has whole first row of convolution kernels completely unchanged during training, and I'm not sure it should be this way. $\endgroup$ – Andrey Jul 1 '16 at 12:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.