# Siamese neural network, why is my gradient descent updating in the wrong direction?

I've been trying to implement a siamese neural network in Torch/Lua, as I already explained here. Now I have my first implementation, that I suppose to be good.

Unfortunately, I'm facing a problem: during training back-propagation, the gradient descent does not update the error in the right direction. That is, often when I have a +1 target goal, the gradient descent goes towards -1, and viceversa (-1 target goal and +1 optimization).

For example, this is my gradient descent for an iteration:

i=1) predictionValue=0.696 target=-1
i=2) predictionValue=0.453 target=-1
i=3) predictionValue=0.999 target=-1
i=4) predictionValue=0.73 target=-1
i=5) predictionValue=0.907 target=-1
i=6) predictionValue=0.545 target=-1
i=7) predictionValue=0.316 target=-1
i=8) predictionValue=0.614 target=-1
i=9) predictionValue=0.999 target=-1
i=10) predictionValue=0.846 target=-1
i=11) predictionValue=1 target=-1
i=12) predictionValue=0.551 target=-1
i=13) predictionValue=0.378 target=-1
i=14) predictionValue=0.407 target=-1
i=15) predictionValue=0.104 target=-1
i=16) predictionValue=0.557 target=-1
i=17) predictionValue=0.65 target=-1
i=18) predictionValue=0.918 target=-1
i=19) predictionValue=0.923 target=-1
i=20) predictionValue=0.882 target=-1
i=21) predictionValue=0.665 target=-1
i=22) predictionValue=0.921 target=-1
i=23) predictionValue=0.969 target=-1
i=24) predictionValue=0.961 target=-1
i=25) predictionValue=0.966 target=-1
i=26) predictionValue=1 target=-1
i=27) predictionValue=0.952 target=-1
i=28) predictionValue=0.966 target=-1
i=29) predictionValue=0.999 target=-1
i=30) predictionValue=1 target=-1


As you can see, the prediction should go towards -1 but, on the contrary, it goes towards +1. Why is this happening?

Here's my working Torch code, that you might try to run (if you've time):

LEARNING_RATE_CONST = 0.1;
output_layer_number = 5;
MAX_ITERATIONS_CONST = 30;

require 'os'
require 'nn'

-- gradient update for the siamese neural network
function gradientUpdate(perceptron, dataset_vector, targetValue, learningRate, max_iterations)

print('gradientUpdate()\n')

for i = 1, max_iterations do

predictionValue = perceptron:forward(dataset_vector)[1]
sys.sleep(0.4)
realTarget=-tonumber(targetValue)

if predictionValue*realTarget < 1 then
gradientWrtOutput = torch.Tensor({realTarget})

perceptron:zeroGradParameters()
perceptron:backward(dataset_vector, gradientWrtOutput)

perceptron:updateParameters(learningRate)

predictionValue = perceptron:forward(dataset_vector)[1]
io.write("i="..i..") predictionValue="..predictionValue.." target="..realTarget.."\n");

if(predictionValue==realTarget) then
io.write("\t@@@ (i="..i..") predictionValue "..predictionValue.." @@@\n");
break
end
end
end
return perceptron;
end

input_number = 6; -- they are 6
dim = 10
hiddenUnits = 3

trueTarget=1; falseTarget=-trueTarget;

trainDataset = {}; targetDataset = {};
for i=1, dim do
trainDataset[i]={torch.rand(input_number),  torch.rand(input_number)}
if i%2==0 then targetDataset[i] = trueTarget
else  targetDataset[i] = falseTarget
end
end

-- imagine we have one network we are interested in, it is called "perceptronUpper"
perceptronUpper= nn.Sequential()
print('input_number='..input_number..'\thiddenUnits='..hiddenUnits);
perceptronUpper:add(nn.Linear(input_number, hiddenUnits))
perceptronUpper:add(nn.Tanh())
if dropOutFlag==TRUE then perceptronUpper:add(nn.Dropout()) end

perceptronUpper:add(nn.Linear(hiddenUnits,output_layer_number))
perceptronUpper:add(nn.Tanh())

perceptronLower = perceptronUpper:clone('weight', 'gradWeight', 'gradBias', 'bias')

parallel_table = nn.ParallelTable()
parallel_table:add(perceptronUpper)
parallel_table:add(perceptronLower)

perceptron= nn.Sequential()
perceptron:add(parallel_table)
perceptron:add(nn.CosineDistance())

max_iterations = MAX_ITERATIONS_CONST;
learnRate = LEARNING_RATE_CONST;

-- # TRAINING:
for k=1, dim do
print('\n[k='..k..'] gradientUpdate()');
perceptron = gradientUpdate(perceptron, trainDataset[k], targetDataset[k], learnRate, max_iterations)
end


Why is my gradient update going in the wrong directions?

• I don't see that it continuously goes into wrong direction, but rather a random wander. Why did you use a randomly generated set for training and test? How can be the objective function minimized if associated labels are random? Jun 15 '15 at 20:09

## 2 Answers

I don't know exactly what your code is doing, but here is my guess, and if so, you have joined a long list of people who have made the same mistake since time immemorial.

In gradient descent for a MINIMIZATION problem, let x be the argument value, and alpha a positive number (perhaps equal to one, but not necessarily), then what you need is something like (see https://en.wikipedia.org/?title=Gradient_descent )

x = x - alpha * (gradient of objective function)


and my guess is that what you have dome is something like

x = x + alpha * (gradient of objective function)


The second of these would be correct if you were doing maximization.

Since you are minimizing, you need to move in the opposite direction of the gradient, because the gradient is in the direction of steepest ascent, i.e., increase in objective function.

Edit: Added per my comment below. One other possibility is that your alpha may be (at least sometimes) too large, and you may be going too far in the correct direction. Perhaps your alpha is always one? I am used to working with algorithms in which the alpha would be adaptively determined based on a line search in the descent direction, or limited in magnitude by use of trust region. Even if you go in a descent direction, if you go too far, the objective function can get worse (increase for minimization problem).

The final possibility I can think of, but no idea if it applies in your case, is that you have computed the gradient incorrectly. This is EXTREMELY common in optimization problems, even by people with Ph.D.s in math, physics, EE, etc.

• Thanks @Mark L. Stone but that's not my case. I'm maximizin' the gradient descent when I want to get the +1 target, and I'm minimizin' the gradient descent when I want to get the -1 target. Jun 15 '15 at 19:07
• I don't understand Siamese Neural Network,and i don't understand your code. However, if you always go the opposite of the correct direction to improve the objective function, then my guess stands. You would also not be the first person to claim they are doing it correctly, when in fact they are doing the opposite. If you write out your algorithm in math notation, perhaps that would help prevent misunderstandings. See my edited answer for an additional possibility. Jun 15 '15 at 19:13
• And just for the sake of clarity, I don't understand what you mean by " I'm maximizin' the gradient descent when I want to get the +1 target, and I'm minimizin' the gradient descent when I want to get the -1 target." That is not proper mathematical language (you don't maximize or minimize "a gradient descent"), and given that it gets right to the heart of where your problem may well be, it is important to write out very clearly and explicitly exactly what you mean. I don't know about Siamese Neural Networks, but i do know about nonlinear optimization, .and certain principle apply regardless. Jun 15 '15 at 19:22
• I wanted to say "I'm maximizin' the gradient descent prediction value when I want to get the +1 target, and I'm minimizin' the gradient descent prediction value when I want to get the -1 target" Jun 15 '15 at 20:52
• That is not a clear explanation. It would help if you wrote out exactly what the algorithm is doing. Regardless of whether you think you are maximizing or minimizing gradient descent prediction (what ever that means), you haven't written out what you are doing, which may not be what you want to be doing. That is the whole point of asking for help, no? Jun 16 '15 at 1:30

Are you correcting your gradients using the curvature as in Newton's method? If so, your update will just as easily go down the gradient to an loss minimum as it will climb up the gradient towards a saddle point.

• He is clearly not using a Newton's method. Even with Newton's method, there is the possibility of going too far along a descent direction, if safeguards such as line search or trust regions are not used. And you need to apply the update in the correct direction. Steepest Descent is basically Newton's method in which the Identity matrix is used as the Hessian (2nd partial derivative matrix). Unlike Newton's method, Steepest Descent always uses a positive definite "Hessian", and if Newton's method has non positive semidefiinite Hessian, additional safeguards are needed. Jun 16 '15 at 1:36
• @MarkL.Stone Yes, my point is that if his curvature matrix might not be positive semidefinite, that would explain moving in a direction opposite to the gradient. I did not read through the code. Jun 16 '15 at 1:37
• He is not using a curvature matrix, except implicitly using the Identity matrix as such, as is always done in steepest (gradient) descent, even though most people never think of it that way. Jun 16 '15 at 1:50