19 questions linked to/from How can change in cost function be positive?
116k views

### Training loss increases with time [duplicate]

I am training a model (Recurrent Neural Network) to classify 4 types of sequences. As I run my training I see the training loss going down until the point where I correctly classify over 90% of the ...
• 453
1 vote
2k views

### Gradient descent decreasing loss [duplicate]

Is the following statement true: Gradient descent is guaranteed to always decrease a loss function. I know that if the loss function is convex, then each iteration of gradient descent will result in ...
• 205
1k views

### Is training loss guaranteed to decrease for stochastic gradient descent? [duplicate]

When performing stochastic gradient descent, it is necessary for the training loss to decrease a) between iterations in an epoch? (I think the answer is no) b) between epochs? (I think the answer is ...
• 1,202
510 views

### PyTorch - Error going up [duplicate]

I'm currently trying to get the basics of Pytorch, playing around with simple networks topologies for the fashion-MNIST dataset. However, when I record the loss of those models after each epochs, it ...
• 23
254 views

### Can Gradient Descent "Bounce Around" Forever? [duplicate]

When learning about Neural Networks and Gradient Descent, we are often shown the following picture that illustrates the obstacles that can be encountered when trying to optimize the Loss Functions ...
1 vote
235 views

### Training loss, validation loss and WER decrease, then increase [duplicate]

I am trying to use Hugginface Datasets for speech recognition using transformers using this tutorial, epochs=30, steps=400, train_batch_size=16. Training loss, validation loss and WER decrease, and ...
• 111
73 views

### What stops gradient descent from finding the largest error? [duplicate]

If a gradient points towards a max or a min what stops gradient descent from maximizing error instead of minimizing it? Is it the nature of the update step that makes this process one way?
67 views

### The result of back propagation for a neural network [duplicate]

I have created a neural network that feeds an image into a convolutional neural net, then feeds the flattened output of this network into an artificial neural network. I have a feeling that my ...
• 33
16 views

### Gradient descent loss increase [duplicate]

I am following the CS231n NN case study — a derivation of gradient descent for a simple network with a single hidden layer. I have followed the rest of the tutorial and have confidence that the ...
• 709
364k views

### What should I do when my neural network doesn't learn?

I'm training a neural network but the training loss doesn't decrease. How can I fix this? I'm not asking about overfitting or regularization. I'm asking about how to solve the problem where my network'...
• 92.5k
11k views

### For convex problems, does gradient in Stochastic Gradient Descent (SGD) always point at the global extreme value?

Given a convex cost function, using SGD for optimization, we will have a gradient (vector) at a certain point during the optimization process. My question is, given the point on the convex, does the ...
• 2,475
10k views

### Why are second-order derivatives useful in convex optimization?

I guess this is a basic question and it has to do with the direction of the gradient itself, but I'm looking for examples where 2nd order methods (e.g. BFGS) are more effective than simple gradient ...
• 2,872
2k views

I am trying to understand gradient descent optimization in ML(machine learning) algorithms. I understand that there's a cost function—where the aim is to minimize the error $\hat y-y$. In a ...
• 345
15k views

### Gradient descent explodes if learning rate is too large

I've implemented my own gradient descent algorithm for an OLS, code below. It work's, however, when the learning rate is too large (i.e. learn_rate >= .3), my approach is unstable. The coefficient's ...
• 922