Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.

961 questions
Filter by
Sorted by
Tagged with
18 views

### Does normalizing/changing the scale of the target variable impact the shape of the loss function equation?

I was under the impression that changing the scale/normalizing the target variable in a regression task would not change the overall shape of the loss function equation but would simply translate/move ...
1 vote
19 views

### Could someone help me interpret the data that I have gathered thus far?

I am trying to train a SVM model for my statistical learning course. The problem is a binary image classification problem (wildfire, nowildfire). This is the rigorous amount of testing that I have ...
• 111
9 views

### Gradient of multiclass hinge loss (max of max difference version)

I want to train a linear classifier for image classification. I have a $\mathbf{W}$ of shape $D\times K$ where $D$ is the dimension of the vectorized version of the image (including bias, herein 3073) ...
• 220
53 views

### How did deep learning overcome numerical problems associated with earlier ANNs? [closed]

From my understanding, the basic design of an artificial neuron has remained essentially the same since the 1960s. Before the bloom of deep learning models in ~2010, there were two obstacles to deep ...
• 101
30 views

### Target binning in regression

I would like to find a predictive density for target variable via multi-class classification. Suppose we are given a set of features $\mathbf X$ and continuous target $\mathbf y$. Replace each $y$ ...
• 509
1 vote
38 views

### Neural network parameters dependency vs gradient descent [duplicate]

Neural network parameters are not independent from each other. How do we account for this dependence in the gradient descent algorithm? Intuitively I would expect that if we first change the weights ...
• 371
24 views

### Why do the error derivatives become small if we start with a large learning rate?

In these slides from Hinton (https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf) there is this statement: I don't understand why "The error derivatives for the hidden units ...
• 1
30 views

### My loss has a non-differentiable point

I had to design a loss function max(0,x). It's not differentiable at x=0. In order to train it with gradient descent, what should I do? I have learned that subgradient can be used instead, so does it ...
1 vote
12 views

• 311
132 views

### Why do we regularize large gradients corresponding to large errors?

While reviewing some scientific blogs, I found them recommending using gradient clipping for large error gradients. However, intuitively one would think that when model predictions are completely off, ...
1 vote