1
$\begingroup$

I had a backpropagation model which worked perfectly fine, however I wanted to implemented batch training.

Code before batch training (in backpropagation function), pseudocode:

forevery(connection in this.connections.in){
  // Adjust weight
  var deltaWeight = rate * gradient + momentum * connection.previousDeltaWeight;
  connection.weight += deltaWeight;
  connection.previousDeltaWeight = deltaWeight;
}

// Adjust bias
var deltaBias = rate * this.error.responsibility + momentum * this.previousDeltaBias;
this.bias += deltaBias;

this.previousDeltabias = deltaBias;

And the new code is:

forevery(connection in this.connections.in){
  // Adjust weight
  var deltaWeight = rate * gradient * this.mask + momentum * connection.previousDeltaWeight;
  connection.totalDeltaWeight += deltaWeight;
  if(update){
    connection.weight += connection.totalDeltaWeight;
    connection.previousDeltaWeight = connection.totalDeltaWeight;
    connection.totalDeltaWeight = 0;
  }
}

// Adjust bias
var deltaBias = rate * this.error.responsibility + momentum * this.previousDeltaBias;
this.totalDeltaBias += deltaBias;
if(update){
  this.bias += this.totalDeltaBias;
  this.previousDeltaBias = this.totalDeltaBias;
  this.totalDeltaBias = 0;
}

So if the batch size is 4, the backpropagation gets called 3x with update=false and the 4th time with update=true. The batch training works fine, but when I turn on momentum (=0.9) all the values start overflowing. What could be the problem?

$\endgroup$
0
$\begingroup$

Wow. I was accumulating the momentum wrong. I included it batch_size times which was wrong, so it is now only included once.

forevery(connection in this.connections.in){
  // Adjust weight
  var deltaWeight = rate * gradient * this.mask;
  connection.totalDeltaWeight += deltaWeight;
  if(update){
    connection.totalDeltaWeight += momentum * connection.previousDeltaWeight;
    connection.weight += connection.totalDeltaWeight;
    connection.previousDeltaWeight = connection.totalDeltaWeight;
    connection.totalDeltaWeight = 0;
  }
}

// note: MINI_BATCH SHALL BE OPTIMIZED SOON

// Adjust bias
var deltaBias = rate * this.error.responsibility;
this.totalDeltaBias += deltaBias;
if(update){
  this.totalDeltaBias += momentum * this.previousDeltaBias;
  this.bias += this.totalDeltaBias;
  this.previousDeltaBias = this.totalDeltaBias;
  this.totalDeltaBias = 0;
}
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.