2
$\begingroup$

I would like to train a simple Artificial Neural Network implementing an algorithm of the class of limited-memory Quasi-Newton.

I read the paper Modified quasi-Newton methods for training neural networks where they train a NN using the BFGS formula. They also proposed to compute the Hessian by layer (BFGS-L), by neuron (BFGS-N) or mixed (BFGS-M), thus decreasing the total number of partial derivative to compute. As I understood, the true Hessian is computed for each neuron. Clearly is a quasi-Newton. Is it limited-memory?

Instead, Limited-memory BFGS approximates the Hessian using the past m iterates and gradients.

  1. Are they both limited-memory quasi-Newton?
  2. Why choose one over the other?
  3. Which one is easier to implement given that I implement the network as a list of layers, each having a list of neurons, and with the weights stored in each neuron?

p.s. maybe this question can be in the Mathematics forum.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.