Linked Questions

1
vote
1answer
121 views

Can we apply analyticity of a neural network to improve upon gradient descent? [duplicate]

Gradient descent uses the first order derivative information of the objective function as a function of the parameters. Gradient descent therefore uses only “local” information about the objective ...
3
votes
0answers
79 views

Second derivative test for machine learning algorithms [duplicate]

I have a question on second derivative test for most "modern" machine learning algorithms. I learned that in calculus but never seen it in real applications. Most machine learning algorithms ...
0
votes
0answers
30 views

Gradient Descent vs Newton's Method? [duplicate]

Can someone please elaborate on Gradient Descent vs Newton's Method? I am studying for my ML exam tomorrow and saw this as a bullet point but cannot find many great answers. Can anyone fill me in some ...
209
votes
8answers
136k views

What should I do when my neural network doesn't learn?

I'm training a neural network but the training loss doesn't decrease. How can I fix this? I'm not asking about overfitting or regularization. I'm asking about how to solve the problem where my ...
28
votes
3answers
9k views

Why use gradient descent with neural networks?

When training a neural network using the back-propagation algorithm, the gradient descent method is used to determine the weight updates. My question is: Rather than using gradient descent method to ...
7
votes
4answers
938 views

Loss function in machine learning - how to constrain?

My loss has two parts, say L1 and L2. I want to minimize both, and at the same time I need to constrain that L1 should be always greater than L2 (L1>L2). Is the following correct? loss = L2 - L1
17
votes
2answers
3k views

Why second order SGD convergence methods are unpopular for deep learning?

It seems that, especially for deep learning, there are dominating very simple methods for optimizing SGD convergence like ADAM - nice overview: http://ruder.io/optimizing-gradient-descent/ They trace ...
1
vote
2answers
1k views

Why is the second derivative required for newton's method for back-propagation?

I am troubled with why isn't the Newton's method used for backpropagation, instead, or in addition to Gradient Descent more widely. I have seen this same question, and the widely accepted answer ...
6
votes
2answers
248 views

Looking for book recommendations for numerical optimization

I was reading the answers and comments to this question: Why is Newton's method not widely used in machine learning? and realised that I would like to learn a lot more about numerical optimization....
3
votes
1answer
852 views

Why do saddle points become “attractive” in Newtonian dynamics?

I am reading Identifying and attacking the saddle point problem in high-dimensional non-convex optimization by Dauphin et. al. and the first paragraph on the second page states the following: A ...
1
vote
1answer
489 views

ADAM Gradient descent oscillates close to minimum

I am using ADAM as an optimization algorithm to minimize some black box function $f(x,y)$. I know this function is convex and has a minimum $f(5,5) = 0$. Initially, the algorithm proceeds as expected:...
1
vote
1answer
233 views

Saddle-free Newton method for SGD - while Newton attracts saddles, is it worth to actively replel them?

While 2nd order methods have many advantages, e.g. natural gradient (e.g. in L-BFGS) attracts to close zero gradient point, which is usually saddle. Other try to pretend that our very non-convex ...
2
votes
0answers
267 views

Is that true Newton's Method / Quasi Newton Method are not widely used in deep neutral network training? [duplicate]

In recent years, people build huge neural networks with millions of parameters to learn. I have seen many discussions about gradient based training, but not too much for Newton's Method / Quasi ...
1
vote
1answer
187 views

Are there are alternatives to gradient update rule?

Most optimization techniques (that I'm aware of) for non-linear cost functions that are commonly implemented rely on linearly updating a variable iteratively until a minimum is reached or a condition ...
1
vote
0answers
65 views

Momentum updates average of $g$, Adagrad also of $g^2$ - any other interesting updated averages for SGD convergence?

Updating exponential moving average is a basic tool of SGD methods, starting with of gradient $g$ in momentum method to extract local linear trend from the statistics. Then e.g. Adagrad, ADAM family ...