# Numerical gradient check of input variable in neural network

I'm trying to numerically check the gradient in a neural network (finite difference approximation), similar to Numerical check of gradient in neural network . However, I want to compute the gradient wrt. the input not the parameters.

As test criterion, I use the relative error of the numerical (finite difference) and analytical (backprop) gradient, defined in https://cs231n.github.io/neural-networks-3/#gradcheck .

Unfortunately, the relative error is very high (~0.9), regardless of the stepsize chosen for the finite difference.

Here's some PyTorch code for a simple linear layer trained with cross-entropy loss:

import torch
from torch.optim import SGD
import torch.nn as nn
import numpy as np

# relative error
def relerr(g1, g2):
g1 = g1.numpy()
g2 = g2.numpy()
return np.abs(g1 - g2) / np.maximum(np.abs(g1), np.abs(g2))

d = 4  # num features
n = 10  # num samples
k = 2  # num classes
nepochs = 10 # num epochs
m = nn.Linear(d, k)

opt = SGD(m.parameters(), lr=1e-3)
crit = nn.CrossEntropyLoss()

# Input samples
x = torch.randn(n, d)

# Targets
t = torch.LongTensor(np.random.randint(k, size=n))

# Train for a short while
for i in range(nepochs):
y = m(x)
loss = crit(y, t)
loss.backward()
if i < nepochs - 1:
opt.step()

print(loss.item())

# Numerical approximation of input grad for several stepsizes eps
for eps in [10 ** (-e) for e in range(0, 9)]:
xp = x.clone()
xp[j, i] += eps
y = m(xp)
lossp = crit(y, t).item()

xn = x.clone()
xn[j, i] -= eps
y = m(xn)
lossn = crit(y, t).item()

grad_num[j, i] = (lossp - lossn) / (2 * eps)


This gives me:

1e+00 8.996676e-01
1e-01 9.000528e-01
1e-02 9.000629e-01
1e-03 9.000670e-01
1e-04 9.012159e-01
1e-05 8.879865e-01
1e-06 7.996843e-01
1e-07 1.000000e+00
1e-08 1.000000e+00


According to https://cs231n.github.io/neural-networks-3/#gradcheck , the relative error should at least be lower than 1e-2.

Trying the same on the output, i.e. the logits (called y in the code), works better although I need to use a relatively large stepsize:

1e-01 2.426800e-04
1e-02 3.202225e-05
1e-03 3.494641e-04
1e-04 3.063642e-03
1e-05 1.855947e-02
1e-06 5.074288e-01


Is there something wrong in my code? Or are there more things to consider when computing the input grad than for the parameters (I can't see why though)?