11
$\begingroup$

I'm training a convolutional neural network (CNN) with 5 conv-layers and 2 fully-connected layers for binary classification using stochastic gradient descent (SGD) with momentum. The accuracy of the network is fine, but I am curious about the behavior of the gradients. I would expect that the 2-norm of the gradients would get lowered during training due to continuous lowering of the learning rate. However, for all of my layers the 2-norm increases slightly during training. How can that be?

More importantly, what should I do to fix it?

Below is an image of the 2-norm of the gradients for my last fully-connected layer (it has 450 neurons). The development for the remaining layers look identical, but the 2-norm is shifted as the gradients are lower for earlier layers (as expected).

enter image description here

Update: This is the code I used to find the gradient of the CNN implemented in Caffe. It should be working correctly based on the answers in this StackOverflow post. The batch-size of my network is 32 and I use leaky ReLU after each layer and dropout after the fully-connected layers.

def find_loss_gradients(self, solver):
    net = solver.net
    diffs = net.backward(diffs=['hzvt1', 'hzvt2', 'hzvt3', 'hzvt4', 'hzvt5', 'hzvt6', 'fc1'])
    l2norms = {}

    for key in diffs.keys():
       grad = diffs[key].flatten()
       l2norms.update({key:np.linalg.norm(grad)})
    return l2norms

def graph_gradients(save_dir, gradient_norm):
    graph_gradient_norm(save_dir + "gradient_all.png", gradient_norm)    
    values_dict = gradient_norm[:,1]
    layers = values_dict[0].keys()
    for key in layers:
        graph_gradient_norm(save_dir + "gradient_{}.png".format(key), gradient_norm, key)

def graph_gradient_norm(save_name, gradient_norm, key=None):
    plt.style.use('ggplot')
    fig = plt.figure()
    _, ax1 = P.subplots()

    x = gradient_norm[:,0]
    values_dict = gradient_norm[:,1]
    handles = []

    if key is None:
        layers = values_dict[0].keys()
    else:
        layers = [key]

    for key in layers:
        val = cm.utils.extract_array_from_dict(values_dict, key, index_into=False)
        h, = ax1.semilogy(x,val)
        handles.append(h)

    ax1.set_xlabel('Iterations')
    ax1.set_ylabel('2-Norm of Gradient')
    plt.legend(handles, layers)
    P.savefig(save_name, bbox_inches='tight')
    plt.close(fig)

Below is a simplified version of my training code

grad_norms = []
for i in range(n_steps):
    solver.step(1)
    grad_norms.append(find_loss_gradients(solver))
grad_norms = np.array(grad_norms)
graph_gradients(run_dir + "visualization/", grad_norms)

Update 2

Note that the loss stops decreasing quite quickly. The same is true for the validation accuracy. I'm currently trying to increase the batch size and see if it has an influence.

$\endgroup$
7
  • 2
    $\begingroup$ That's an interesting point! Which algorithm/solver are you using? $\endgroup$
    – RUser4512
    Sep 23, 2015 at 11:49
  • $\begingroup$ SGD with momentum :) $\endgroup$
    – pir
    Sep 23, 2015 at 12:16
  • 2
    $\begingroup$ Just to clarify. Are you 100% sure that this is a gradient and not a $ \Delta w $ 2-norm value? $\endgroup$
    – itdxer
    Sep 23, 2015 at 12:35
  • $\begingroup$ Well, I've found it based on the answer by cesans in stackoverflow.com/questions/31324739/…. See my updated post. $\endgroup$
    – pir
    Sep 23, 2015 at 14:25
  • $\begingroup$ Can you test it with less number of iterations? Maybe hundred epochs would be enough. $\endgroup$
    – itdxer
    Sep 23, 2015 at 19:48

2 Answers 2

10
$\begingroup$

I guess I got what is a problem with a gradient norm value. Basically negative gradient shows a direction to a local minimum value, but it doesn't say how far it is. For this reason you are able to configure you step proportion. When your weight combination is closer to the minimum value your constant step could be bigger than is necessary and some times it hits in wrong direction and in next epooch network try to solve this problem. Momentum algorithm use modified approach. After each iteration it increases weight update if sign for the gradient the same (by an additional parameter that is added to the $\Delta w$ value). In terms of vectors this addition operation can increase magnitude of the vector and change it direction as well, so you are able to miss perfect step even more. To fix this problem network sometimes needs a bigger vector, because minimum value a little further than in the previous epoch.

To prove that theory I build small experiment. First of all I reproduce the same behaviour but for simpler network architecture with less number of iterations.

import numpy as np
from numpy.linalg import norm
import matplotlib.pyplot as plt
from sklearn.datasets import make_regression
from sklearn import preprocessing
from sklearn.pipeline import Pipeline
from neupy import algorithms

plt.style.use('ggplot')

grad_norm = []
def train_epoch_end_signal(network):
    global grad_norm
    # Get gradient for the last layer
    grad_norm.append(norm(network.gradients[-1]))

data, target = make_regression(n_samples=10000, n_features=50, n_targets=1)

target_scaler = preprocessing.MinMaxScaler()
target = target_scaler.fit_transform(target)

mnet = Pipeline([
    ('scaler', preprocessing.MinMaxScaler()),
    ('momentum', algorithms.Momentum(
        (50, 30, 1),
        step=1e-10,
        show_epoch=1,
        shuffle_data=True,
        verbose=False,
        train_epoch_end_signal=train_epoch_end_signal,
    )),
])

mnet.fit(data, target, momentum__epochs=100)

After training I checked all gradients on plot. Below you can see similar behaviour as yours.

plt.figure(figsize=(12, 8))
plt.plot(grad_norm)
plt.title("Momentum algorithm final layer gradient 2-Norm")
plt.ylabel("Gradient 2-Norm")
plt.xlabel("Epoch")
plt.show()

Momentum algorithm final layer gradient 2-Norm

Also if look closer into the training procedure results after each epoch you will find that errors are vary as well.

plt.figure(figsize=(12, 8))
network = mnet.steps[-1][1]
network.plot_errors()
plt.show()

Train errors

Next I using almost the same settings create another network, but for this time I select Golden search algorithm for step selection on each epoch.

grad_norm = []
def train_epoch_end_signal(network):
    global grad_norm
    # Get gradient for the last layer
    grad_norm.append(norm(network.gradients[-1]))
    if network.epoch % 20 == 0:
        print("Epoch #{}: step = {}".format(network.epoch, network.step))

mnet = Pipeline([
    ('scaler', preprocessing.MinMaxScaler()),
    ('momentum', algorithms.Momentum(
        (50, 30, 1),
        step=1e-10,
        show_epoch=1,
        shuffle_data=True,
        verbose=False,
        train_epoch_end_signal=train_epoch_end_signal,
        optimizations=[algorithms.LinearSearch]
    )),
])

mnet.fit(data, target, momentum__epochs=100)

Output below shows step variation at each 20 epoch.

Epoch #0: step = 0.5278640466583575
Epoch #20: step = 1.103484809236065e-13
Epoch #40: step = 0.01315561773591515
Epoch #60: step = 0.018180616551587894
Epoch #80: step = 0.00547810271094794

And if you after that training look closer into the results you will find that variation in 2-norm is much smaller

plt.figure(figsize=(12, 8))
plt.plot(grad_norm)
plt.title("Momentum algorithm final layer gradient 2-Norm")
plt.ylabel("Gradient 2-Norm")
plt.xlabel("Epoch")
plt.show()

Momentum algorithm with Golden search, final layer gradient 2-Norm

And also this optimization reduce variation of errors as well

plt.figure(figsize=(12, 8))
network = mnet.steps[-1][1]
network.plot_errors()
plt.show()

Train errors

As you can see the main problem with gradient is in the step length.

It's important to note that even with a high variation your network can give you improve in your prediction accuracy after each iteration.

$\endgroup$
5
  • $\begingroup$ Thanks, it looks really cool! So based on your work here your recommendation would be to use smaller step size and decrease variation if possible? $\endgroup$
    – pir
    Sep 26, 2015 at 22:01
  • $\begingroup$ Variation in gradient 2-Norm isn't so bad, if your error decrease during the training procedure. Usually, if your problem is not very big, you can use some efficient step selection algorithms (like the Golden search method), but for the big problems it could be very expensive. Even with a small variation to get a good result it takes more time. $\endgroup$
    – itdxer
    Sep 26, 2015 at 23:27
  • 1
    $\begingroup$ so your answer seems to suggest that its just a coincidence. I always thought this was one of the puzzling behaviours of CNNs. See figure 8 the standard deep learning reference/book: deeplearningbook.org/contents/optimization.html $\endgroup$ Aug 24, 2017 at 21:24
  • $\begingroup$ it wasn't a coincidence. the problem occured, because learning rate (aka step size) became too large for accurate updates after a few iterations. From the code you can see that decreasing step size over the time helps resolving the problem $\endgroup$
    – itdxer
    Aug 25, 2017 at 7:59
  • $\begingroup$ Let's say the optimum is at zero. At step t the weight was at +10. Are you saying that the learning rate is maybe too high, so the delta weight is too big, and at step t+1, the weight is at -15 (so even further from the optimum)? And since gradients grow with distance to the optimum, the gradient is even larger now, and next time the weight jumps to +20? And so on? This mechanism would cause the network's accuracy to get worse. Since in the OP question, the accuracy was improving, I don't think this mechanism is the right explanation. Please correct me if I'm wrong. $\endgroup$
    – max
    Jul 1, 2020 at 8:11
6
$\begingroup$

This is normal situation in training of convolutional NNN. Your loss function decrease, but loss gradients go up. This is true both for Vanilla SGD, and for SGD with momentum. The reason for this is that initial learning rate is too high for some layers. There is very good Ian Goodfellow' tutorial on Optimization of NN, where he explains why this happens: http://videolectures.net/deeplearning2015_goodfellow_network_optimization/ , minute 29

$\endgroup$
2
  • $\begingroup$ Thanks! However, see my updated post. My loss and accuracy doesn't continue to improve. $\endgroup$
    – pir
    Sep 30, 2015 at 12:50
  • 1
    $\begingroup$ maybe its just me but I saw that talk multiple times where you suggested and he didn't explain explicitly why nrom gradient decrease despite the loss having converged. Personally, I'm still very confused about this behaviour and can't wrap my head around it. $\endgroup$ Aug 24, 2017 at 21:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.