I would like to train a simple Artificial Neural Network implementing an algorithm of the class of limited-memory Quasi-Newton.

I read the paper Modified quasi-Newton methods for training neural networks where they train a NN using the BFGS formula. They also proposed to compute the Hessian by layer (BFGS-L), by neuron (BFGS-N) or mixed (BFGS-M), thus decreasing the total number of partial derivative to compute. As I understood, the true Hessian is computed for each neuron. Clearly is a quasi-Newton. Is it limited-memory?

Instead, Limited-memory BFGS approximates the Hessian using the past m iterates and gradients.

  1. Are they both limited-memory quasi-Newton?
  2. Why choose one over the other?
  3. Which one is easier to implement given that I implement the network as a list of layers, each having a list of neurons, and with the weights stored in each neuron?

p.s. maybe this question can be in the Mathematics forum.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.