I'm trying to numerically check the gradient in a neural network (finite difference approximation), similar to Numerical check of gradient in neural network . However, I want to compute the gradient wrt. the input not the parameters.

As test criterion, I use the relative error of the numerical (finite difference) and analytical (backprop) gradient, defined in https://cs231n.github.io/neural-networks-3/#gradcheck .

Unfortunately, the relative error is very high (~0.9), regardless of the stepsize chosen for the finite difference.

Here's some PyTorch code for a simple linear layer trained with cross-entropy loss:

import torch
from torch.optim import SGD
import torch.nn as nn
import numpy as np

# relative error
def relerr(g1, g2):
    g1 = g1.numpy()
    g2 = g2.numpy()
    return np.abs(g1 - g2) / np.maximum(np.abs(g1), np.abs(g2))

d = 4  # num features
n = 10  # num samples
k = 2  # num classes
nepochs = 10 # num epochs
m = nn.Linear(d, k)

opt = SGD(m.parameters(), lr=1e-3)
crit = nn.CrossEntropyLoss()

# Input samples
x = torch.randn(n, d)
x.requires_grad = True

# Targets
t = torch.LongTensor(np.random.randint(k, size=n))

# Train for a short while
for i in range(nepochs):
    y = m(x)
    loss = crit(y, t)
    if i < nepochs - 1:


# Analytical gradient
grad_anal_x = x.grad
grad_anal_y = y.grad

# Numerical approximation of input grad for several stepsizes eps
for eps in [10 ** (-e) for e in range(0, 9)]:
    grad_num = torch.zeros_like(x)
    for j in range(grad_num.shape[0]):
        for i in range(grad_num.shape[1]):
            xp = x.clone()
            xp[j, i] += eps
            y = m(xp) 
            lossp = crit(y, t).item()

            xn = x.clone()
            xn[j, i] -= eps
            y = m(xn)
            lossn = crit(y, t).item()

            grad_num[j, i] = (lossp - lossn) / (2 * eps)
    print(f"{eps:.0e}", f"{relerr(grad_num, grad_anal_x).mean():e}")

This gives me:

1e+00 8.996676e-01
1e-01 9.000528e-01
1e-02 9.000629e-01
1e-03 9.000670e-01
1e-04 9.012159e-01
1e-05 8.879865e-01
1e-06 7.996843e-01
1e-07 1.000000e+00
1e-08 1.000000e+00

According to https://cs231n.github.io/neural-networks-3/#gradcheck , the relative error should at least be lower than 1e-2.

Trying the same on the output, i.e. the logits (called y in the code), works better although I need to use a relatively large stepsize:

1e-01 2.426800e-04
1e-02 3.202225e-05
1e-03 3.494641e-04
1e-04 3.063642e-03
1e-05 1.855947e-02
1e-06 5.074288e-01

Is there something wrong in my code? Or are there more things to consider when computing the input grad than for the parameters (I can't see why though)?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.