Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.

710 questions
Filter by
Sorted by
Tagged with
35 views

### Why do earlier hidden layers learn slower?

I'm reading chapter 5 of Nielsen's textbook about vanishing gradients. He states: In at least some deep neural networks, the gradient tends to get smaller as we move backward through the hidden ...
4 views

### gradient computation in Nesterov momentum

I am reading deeplearningbook by Ian Goodfellow et al. And I have a question about gradient computation in Nesterov momentum. There are two pages from this book, which describe Nesterov momentum: ...
15 views

### Investigator efficiency computation

I need a little help with inferring ground truth and investigation capabilities from data. I have an incomplete matrix consisting of binary decisions taken by investigators on documents i.e. for every ...
24 views

### Valid use of differentiable almost everywhere functions, like hinge loss, in gradient optimization/learners, like SciKit-Learn's SGDClassifier?

So, my abstract question is: is it valid (in the sense that stable convergence is roughly expected) to use functions that are differentiable almost everywhere in the practical application of ...
12 views

44 views

### How does the batch size affect the Stochastic Gradient Descent optimizer? (Example using Keras)

First of all, I know that there are lots of questions and answers about the topic throughout the site $-$ such as here, here or here (and I've probably read them all). However, I am still confused. ...
18 views

### How to minimize the sum of Frobenius norm and Nuclear norm

I have to minimize an objective function of the the form : $||X_{s} - Y_{s}D_{s}||_{F}^{2} + ||D_{s}||_{F}^{2} + ||D_{s}||_{*}^{2}$ where $||.||_{F}$ denotes the Frobenius norm and $||.||_{*}$ ...