0
$\begingroup$

I'm trying to numerically check the gradient in a neural network (finite difference approximation), similar to Numerical check of gradient in neural network . However, I want to compute the gradient wrt. the input not the parameters.

As test criterion, I use the relative error of the numerical (finite difference) and analytical (backprop) gradient, defined in https://cs231n.github.io/neural-networks-3/#gradcheck .

Unfortunately, the relative error is very high (~0.9), regardless of the stepsize chosen for the finite difference.

Here's some PyTorch code for a simple linear layer trained with cross-entropy loss:

import torch
from torch.optim import SGD
import torch.nn as nn
import numpy as np

# relative error
def relerr(g1, g2):
    g1 = g1.numpy()
    g2 = g2.numpy()
    return np.abs(g1 - g2) / np.maximum(np.abs(g1), np.abs(g2))

d = 4  # num features
n = 10  # num samples
k = 2  # num classes
nepochs = 10 # num epochs
m = nn.Linear(d, k)

opt = SGD(m.parameters(), lr=1e-3)
crit = nn.CrossEntropyLoss()

# Input samples
x = torch.randn(n, d)
x.requires_grad = True

# Targets
t = torch.LongTensor(np.random.randint(k, size=n))

# Train for a short while
for i in range(nepochs):
    opt.zero_grad()
    y = m(x)
    loss = crit(y, t)
    y.retain_grad()
    loss.backward()
    if i < nepochs - 1:
        opt.step()

print(loss.item())

# Analytical gradient
grad_anal_x = x.grad
grad_anal_y = y.grad

# Numerical approximation of input grad for several stepsizes eps
for eps in [10 ** (-e) for e in range(0, 9)]:
    grad_num = torch.zeros_like(x)
    for j in range(grad_num.shape[0]):
        for i in range(grad_num.shape[1]):
            xp = x.clone()
            xp[j, i] += eps
            y = m(xp) 
            lossp = crit(y, t).item()

            xn = x.clone()
            xn[j, i] -= eps
            y = m(xn)
            lossn = crit(y, t).item()

            grad_num[j, i] = (lossp - lossn) / (2 * eps)
    print(f"{eps:.0e}", f"{relerr(grad_num, grad_anal_x).mean():e}")

This gives me:

1e+00 8.996676e-01
1e-01 9.000528e-01
1e-02 9.000629e-01
1e-03 9.000670e-01
1e-04 9.012159e-01
1e-05 8.879865e-01
1e-06 7.996843e-01
1e-07 1.000000e+00
1e-08 1.000000e+00

According to https://cs231n.github.io/neural-networks-3/#gradcheck , the relative error should at least be lower than 1e-2.

Trying the same on the output, i.e. the logits (called y in the code), works better although I need to use a relatively large stepsize:

1e-01 2.426800e-04
1e-02 3.202225e-05
1e-03 3.494641e-04
1e-04 3.063642e-03
1e-05 1.855947e-02
1e-06 5.074288e-01

Is there something wrong in my code? Or are there more things to consider when computing the input grad than for the parameters (I can't see why though)?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.