I would like to train a simple Artificial Neural Network implementing an algorithm of the class of limited-memory Quasi-Newton.
I read the paper Modified quasi-Newton methods for training neural networks where they train a NN using the BFGS formula. They also proposed to compute the Hessian by layer (BFGS-L), by neuron (BFGS-N) or mixed (BFGS-M), thus decreasing the total number of partial derivative to compute. As I understood, the true Hessian is computed for each neuron. Clearly is a quasi-Newton. Is it limited-memory?
Instead, Limited-memory BFGS approximates the Hessian using the past m iterates and gradients.
- Are they both limited-memory quasi-Newton?
- Why choose one over the other?
- Which one is easier to implement given that I implement the network as a list of layers, each having a list of neurons, and with the weights stored in each neuron?
p.s. maybe this question can be in the Mathematics forum.