Siamese neural network, why is my gradient descent updating in the wrong direction?

Question

I've been trying to implement a siamese neural network in Torch/Lua, as I already explained here. Now I have my first implementation, that I suppose to be good.

Unfortunately, I'm facing a problem: during training back-propagation, the gradient descent does not update the error in the right direction. That is, often when I have a +1 target goal, the gradient descent goes towards -1, and viceversa (-1 target goal and +1 optimization).

For example, this is my gradient descent for an iteration:

i=1) predictionValue=0.696 target=-1
i=2) predictionValue=0.453 target=-1
i=3) predictionValue=0.999 target=-1
i=4) predictionValue=0.73 target=-1
i=5) predictionValue=0.907 target=-1
i=6) predictionValue=0.545 target=-1
i=7) predictionValue=0.316 target=-1
i=8) predictionValue=0.614 target=-1
i=9) predictionValue=0.999 target=-1
i=10) predictionValue=0.846 target=-1
i=11) predictionValue=1 target=-1
i=12) predictionValue=0.551 target=-1
i=13) predictionValue=0.378 target=-1
i=14) predictionValue=0.407 target=-1
i=15) predictionValue=0.104 target=-1
i=16) predictionValue=0.557 target=-1
i=17) predictionValue=0.65 target=-1
i=18) predictionValue=0.918 target=-1
i=19) predictionValue=0.923 target=-1
i=20) predictionValue=0.882 target=-1
i=21) predictionValue=0.665 target=-1
i=22) predictionValue=0.921 target=-1
i=23) predictionValue=0.969 target=-1
i=24) predictionValue=0.961 target=-1
i=25) predictionValue=0.966 target=-1
i=26) predictionValue=1 target=-1
i=27) predictionValue=0.952 target=-1
i=28) predictionValue=0.966 target=-1
i=29) predictionValue=0.999 target=-1
i=30) predictionValue=1 target=-1

As you can see, the prediction should go towards -1 but, on the contrary, it goes towards +1. Why is this happening?

Here's my working Torch code, that you might try to run (if you've time):

LEARNING_RATE_CONST = 0.1;
output_layer_number = 5;
MAX_ITERATIONS_CONST = 30;

require 'os'
require 'nn'

-- gradient update for the siamese neural network
function gradientUpdate(perceptron, dataset_vector, targetValue, learningRate, max_iterations)

  print('gradientUpdate()\n')

  for i = 1, max_iterations do

      predictionValue = perceptron:forward(dataset_vector)[1]
      sys.sleep(0.4)
      realTarget=-tonumber(targetValue)

      if predictionValue*realTarget < 1 then
    gradientWrtOutput = torch.Tensor({realTarget})

    perceptron:zeroGradParameters()
    perceptron:backward(dataset_vector, gradientWrtOutput)

    perceptron:updateParameters(learningRate)

    predictionValue = perceptron:forward(dataset_vector)[1]
    io.write("i="..i..") predictionValue="..predictionValue.." target="..realTarget.."\n");

    if(predictionValue==realTarget) then
        io.write("\t@@@ (i="..i..") predictionValue "..predictionValue.." @@@\n");
        break
      end
      end      
    end
  return perceptron;
end

input_number = 6; -- they are 6
dim = 10
hiddenUnits = 3

trueTarget=1; falseTarget=-trueTarget; 

trainDataset = {}; targetDataset = {};
for i=1, dim do
     trainDataset[i]={torch.rand(input_number),  torch.rand(input_number)}
     if i%2==0 then targetDataset[i] = trueTarget
     else  targetDataset[i] = falseTarget 
     end
end


-- imagine we have one network we are interested in, it is called "perceptronUpper"
perceptronUpper= nn.Sequential()
print('input_number='..input_number..'\thiddenUnits='..hiddenUnits);
perceptronUpper:add(nn.Linear(input_number, hiddenUnits))
perceptronUpper:add(nn.Tanh())
if dropOutFlag==TRUE then perceptronUpper:add(nn.Dropout()) end

perceptronUpper:add(nn.Linear(hiddenUnits,output_layer_number))
perceptronUpper:add(nn.Tanh())

perceptronLower = perceptronUpper:clone('weight', 'gradWeight', 'gradBias', 'bias')

parallel_table = nn.ParallelTable()
parallel_table:add(perceptronUpper)
parallel_table:add(perceptronLower)

perceptron= nn.Sequential()
perceptron:add(parallel_table)
perceptron:add(nn.CosineDistance())

max_iterations = MAX_ITERATIONS_CONST;
learnRate = LEARNING_RATE_CONST;


-- # TRAINING:
for k=1, dim do
  print('\n[k='..k..'] gradientUpdate()');
  perceptron = gradientUpdate(perceptron, trainDataset[k], targetDataset[k], learnRate, max_iterations)
  end

Why is my gradient update going in the wrong directions?

Answer 1

I don't know exactly what your code is doing, but here is my guess, and if so, you have joined a long list of people who have made the same mistake since time immemorial.

In gradient descent for a MINIMIZATION problem, let x be the argument value, and alpha a positive number (perhaps equal to one, but not necessarily), then what you need is something like (see https://en.wikipedia.org/?title=Gradient_descent )

x = x - alpha * (gradient of objective function)

and my guess is that what you have dome is something like

x = x + alpha * (gradient of objective function)

The second of these would be correct if you were doing maximization.

Since you are minimizing, you need to move in the opposite direction of the gradient, because the gradient is in the direction of steepest ascent, i.e., increase in objective function.

Edit: Added per my comment below. One other possibility is that your alpha may be (at least sometimes) too large, and you may be going too far in the correct direction. Perhaps your alpha is always one? I am used to working with algorithms in which the alpha would be adaptively determined based on a line search in the descent direction, or limited in magnitude by use of trust region. Even if you go in a descent direction, if you go too far, the objective function can get worse (increase for minimization problem).

The final possibility I can think of, but no idea if it applies in your case, is that you have computed the gradient incorrectly. This is EXTREMELY common in optimization problems, even by people with Ph.D.s in math, physics, EE, etc.

Answer 2

Are you correcting your gradients using the curvature as in Newton's method? If so, your update will just as easily go down the gradient to an loss minimum as it will climb up the gradient towards a saddle point.