Linked Questions
19 questions linked to/from How can change in cost function be positive?
45
votes
3
answers
116k
views
Training loss increases with time [duplicate]
I am training a model (Recurrent Neural Network) to classify 4 types of sequences. As I run my training I see the training loss going down until the point where I correctly classify over 90% of the ...
- 453
1
vote
1
answer
2k
views
Gradient descent decreasing loss [duplicate]
Is the following statement true: Gradient descent is guaranteed to always decrease a loss function.
I know that if the loss function is convex, then each iteration of gradient descent will result in ...
- 205
0
votes
1
answer
1k
views
Is training loss guaranteed to decrease for stochastic gradient descent? [duplicate]
When performing stochastic gradient descent, it is necessary for the training loss to decrease
a) between iterations in an epoch? (I think the answer is no)
b) between epochs? (I think the answer is ...
- 1,202
0
votes
1
answer
510
views
PyTorch - Error going up [duplicate]
I'm currently trying to get the basics of Pytorch, playing around with simple networks topologies for the fashion-MNIST dataset.
However, when I record the loss of those models after each epochs, it ...
- 23
2
votes
0
answers
254
views
Can Gradient Descent "Bounce Around" Forever? [duplicate]
When learning about Neural Networks and Gradient Descent, we are often shown the following picture that illustrates the obstacles that can be encountered when trying to optimize the Loss Functions ...
1
vote
0
answers
235
views
Training loss, validation loss and WER decrease, then increase [duplicate]
I am trying to use Hugginface Datasets for speech recognition using transformers using this tutorial, epochs=30, steps=400, train_batch_size=16. Training loss, validation loss and WER decrease, and ...
- 111
0
votes
1
answer
73
views
What stops gradient descent from finding the largest error? [duplicate]
If a gradient points towards a max or a min what stops gradient descent from maximizing error instead of minimizing it?
Is it the nature of the update step that makes this process one way?
0
votes
0
answers
67
views
The result of back propagation for a neural network [duplicate]
I have created a neural network that feeds an image into a convolutional neural net, then feeds the flattened output of this network into an artificial neural network.
I have a feeling that my ...
- 33
0
votes
0
answers
16
views
Gradient descent loss increase [duplicate]
I am following the CS231n NN case study — a derivation of gradient descent for a simple network with a single hidden layer.
I have followed the rest of the tutorial and have confidence that the ...
- 709
366
votes
9
answers
364k
views
What should I do when my neural network doesn't learn?
I'm training a neural network but the training loss doesn't decrease. How can I fix this?
I'm not asking about overfitting or regularization. I'm asking about how to solve the problem where my network'...
- 92.5k
30
votes
6
answers
11k
views
For convex problems, does gradient in Stochastic Gradient Descent (SGD) always point at the global extreme value?
Given a convex cost function, using SGD for optimization, we will have a gradient (vector) at a certain point during the optimization process.
My question is, given the point on the convex, does the ...
- 2,475
25
votes
4
answers
10k
views
Why are second-order derivatives useful in convex optimization?
I guess this is a basic question and it has to do with the direction of the gradient itself, but I'm looking for examples where 2nd order methods (e.g. BFGS) are more effective than simple gradient ...
- 2,872
12
votes
4
answers
2k
views
Gradient descent optimization
I am trying to understand gradient descent optimization in ML(machine learning) algorithms. I understand that there's a cost function—where the aim is to minimize the error $\hat y-y$. In a ...
- 345
6
votes
2
answers
15k
views
Gradient descent explodes if learning rate is too large
I've implemented my own gradient descent algorithm for an OLS, code below. It work's, however, when the learning rate is too large (i.e. learn_rate >= .3), my approach is unstable. The coefficient's ...
- 922
10
votes
2
answers
8k
views
What is the cause of the sudden drop in error rate that one often sees when training a CNN? [duplicate]
I've noticed in different papers that after a certain number of epochs there sometimes is a sudden drop in error rate when training a CNN.
This example is taken from the "Densely Connected ...
- 101