I'm following Google's TensorFlow Deep MNIST for Experts tutorial.
Here is my code:
The networks seems to get close to 100% accuracy after about training 1000 steps, but all of its kernels on the outermost layer remain completely unchanged.
Here's a visualization I made: http://i.imgur.com/gCmxq40.png Each row represents one of 32 5x5 kernels defined by W_conv1; leftmost image in the row is initial state of the kernel, and each next image horizontally is how that kernel looks after 5 training steps.
As it can be clearly seen from the picture, kernels seem to be completely static.
I used the same code to draw kernels for a shallow network with convolutions, and they train very well - you can see shapes slowly forming out of noise. But on that particular example, kernels stay completely same.
Is this normal? Is this a flaw in google's example of deep CNN? Is this some mistake I made following the tutorial?