Questions tagged [stochastic-gradient-descent]

59 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
0
votes
0answers
21 views

Can a neural network still manage to converge, with slightly incorrect gradients?

In a network, we find gradients of the error function w.r.t each of the parameters used in the network. We then update the weights say, using vanilla Gradient Descent. If the computed gradients, do ...
0
votes
0answers
15 views

can alternating optimization be performed in mini-batches

Just wondering if alternating minimization could be performed in mini-batches (just like we have gradient descent and its mini-batch version). Although I am perfectly fine with the full batch version ...
0
votes
0answers
60 views

Why not use line search in conjunction with stochastic gradient descent?

I'm familiar with numerical optimization in Engineering context. I have taken several graduate level engineering optimization and operations research courses. I'm beginning to learn machine learning. ...
0
votes
0answers
7 views

Difference between using SGD on new data in batches vs training on the whole dataset (recommender systems)

I work with recommender systems and use SGD to train them. I am doing both: real-time updates: updating weights as soon as a new batch of 64 entries come in and training on the whole (well part of ...
0
votes
0answers
10 views

High initial test validation score for Neural Network

From only the first epoch of training my NN with SGD (I use Xavier initialisation for weights), the accuracy shoots off to 92%, and then flattens out. The same thing happens with loss (but lower, of ...
0
votes
0answers
25 views

SGD versus Adamax on XOR operator

I am trying to resolve the xor operator using neural networks, and to accomplish that this is my code: ...
0
votes
1answer
274 views

How to deal with numeric instability in stochastic gradient descent?

Imagine that we try to perform sgd using a gradient that takes very small or very large values (e.g. it is a product of many terms that are larger than 1). Is there a standard approach to deal with ...
0
votes
0answers
104 views

Is the regularization term necessary when classifying one feature?

I'm using the Stochastic Gradient Descent linear classifier (implemented in Scikit-learn) to classify an image pixel by pixel. So my dataset has only one feature, ...
0
votes
1answer
113 views

Stochastic gradient descent update

Equation 93 of Chapter 3 of Michael Nielsen's neural networks book describes the stochastic gradient descent update rule as the following: $w \leftarrow (1-\frac{\eta\lambda}{n})w - \frac{\eta}{m}\...

1
2