I'm training a convolutional neural network (CNN) with 5 conv-layers and 2 fully-connected layers for binary classification using stochastic gradient descent (SGD) with momentum. The accuracy of the network is fine, but I am curious about the behavior of the gradients. I would expect that the 2-norm of the gradients would get lowered during training due to continuous lowering of the learning rate. However, for all of my layers the 2-norm increases slightly during training. How can that be?
More importantly, what should I do to fix it?
Below is an image of the 2-norm of the gradients for my last fully-connected layer (it has 450 neurons). The development for the remaining layers look identical, but the 2-norm is shifted as the gradients are lower for earlier layers (as expected).
Update: This is the code I used to find the gradient of the CNN implemented in Caffe. It should be working correctly based on the answers in this StackOverflow post. The batch-size of my network is 32 and I use leaky ReLU after each layer and dropout after the fully-connected layers.
def find_loss_gradients(self, solver):
net = solver.net
diffs = net.backward(diffs=['hzvt1', 'hzvt2', 'hzvt3', 'hzvt4', 'hzvt5', 'hzvt6', 'fc1'])
l2norms = {}
for key in diffs.keys():
grad = diffs[key].flatten()
l2norms.update({key:np.linalg.norm(grad)})
return l2norms
def graph_gradients(save_dir, gradient_norm):
graph_gradient_norm(save_dir + "gradient_all.png", gradient_norm)
values_dict = gradient_norm[:,1]
layers = values_dict[0].keys()
for key in layers:
graph_gradient_norm(save_dir + "gradient_{}.png".format(key), gradient_norm, key)
def graph_gradient_norm(save_name, gradient_norm, key=None):
plt.style.use('ggplot')
fig = plt.figure()
_, ax1 = P.subplots()
x = gradient_norm[:,0]
values_dict = gradient_norm[:,1]
handles = []
if key is None:
layers = values_dict[0].keys()
else:
layers = [key]
for key in layers:
val = cm.utils.extract_array_from_dict(values_dict, key, index_into=False)
h, = ax1.semilogy(x,val)
handles.append(h)
ax1.set_xlabel('Iterations')
ax1.set_ylabel('2-Norm of Gradient')
plt.legend(handles, layers)
P.savefig(save_name, bbox_inches='tight')
plt.close(fig)
Below is a simplified version of my training code
grad_norms = []
for i in range(n_steps):
solver.step(1)
grad_norms.append(find_loss_gradients(solver))
grad_norms = np.array(grad_norms)
graph_gradients(run_dir + "visualization/", grad_norms)
Update 2
Note that the loss stops decreasing quite quickly. The same is true for the validation accuracy. I'm currently trying to increase the batch size and see if it has an influence.