Questions tagged [stochastic-gradient-descent]

Filter by
Sorted by
Tagged with
1
vote
1answer
37 views

SGD unbiased estimator: 1 example vs larger minibatch for each iteration

Studying the SGD, I found that at each iteration the SGD turns out to be an unbiased estimator of the full gradients. The number of iterations (stochastic gradient estimation) depends on the variance. ...
0
votes
0answers
13 views

When is a high learning rate for Stochastic Gradient Descent a good thing?

I was always under the impression that SGD needed a lower learning rate than optimizers like Adam, because it was stochastic and more likely to make training diverge with higher learning rates. I ...
0
votes
0answers
14 views

Reinforcement Learning: SGD sampling and the independence of samples in sequences

I am taking a course in RL and many times, learning policy parameters of value function weights essentially boils down to using Stochastic Gradient Descent (SGD). The agent is represented as having a ...
0
votes
0answers
10 views

Why do we need more than 1 epoch to train data? [duplicate]

As 1 epoch means each data point has gone through the algorithm once and made changes in weighted values accordingly . So , why there is a need to process same data again and again ? How does it ...
2
votes
1answer
48 views

SGD is sensitive to feature scaling

I am taking a deep learning class and the class slides state one of SGD's problems as: "Gradient is scaled equally across all dimensions." Now what is meant by this is I believe, when we ...
2
votes
0answers
20 views

Why is Newton’s method not more common in large scale machine learning if SGD requires grid searching more parameters? [duplicate]

Grid searching might not give the best results. SGD is fast but it’s not going to be as accurate as Newton which directly gives the step size. In SGD you have to find the optimal step size using cross ...
1
vote
1answer
87 views

Adam (adaptive) optimizer(s) learning rate tuning

I'm reading Hands-On Machine Learning with Scikit-Learn, Keras & Tensorflow and on page 325 (follows up on 326) there's a following piece of text on learning-rate: The learning is arguably the ...
6
votes
1answer
201 views

SGD for Gaussian Process estimation

Given a Gaussian process with kernel function $K_{\theta}$ depending on some hyperparameters $\theta$ and set of observations $\{(x_i,y_i)\}_{i=1}^n$, I want to choose $\theta$ to maximize the ...
0
votes
0answers
21 views

Can a neural network still manage to converge, with slightly incorrect gradients?

In a network, we find gradients of the error function w.r.t each of the parameters used in the network. We then update the weights say, using vanilla Gradient Descent. If the computed gradients, do ...
1
vote
1answer
50 views

Optimization of Linear Autoencoder with SGD

I'm interested in the Linear Autoencoder(LAE), and I knew that, at convergence point, the subspace LAE learns is the same as the subspace PCA learns up to linear transformations. Also, the loss ...
0
votes
0answers
9 views

can alternating optimization be performed in mini-batches

Just wondering if alternating minimization could be performed in mini-batches (just like we have gradient descent and its mini-batch version). Although I am perfectly fine with the full batch version ...
3
votes
1answer
71 views

What is the probability distribution of a minibatch of data?

Suppose there are numbers $\{1, \ldots, 10\}$ You pick one at random, call it $i$ Then $i$ is a Uniform random variable (https://en.wikipedia.org/wiki/Discrete_uniform_distribution), $i \sim U\{1, 10\}...
2
votes
1answer
13 views

can we perform sub-gradient updates in mini-batches

We are already aware that in case the data is quite bulky, mini-batch gradient descent based approaches may be applied. These approaches load a mini-batch of data, compute the loss on this batch, and ...
0
votes
1answer
17 views

Anybody know of good material or videos to help me understand Stochastic Gradient Descent?

I am trying to understand stochastic gradient descent a bit better as I'm not 100. Does anybody have any materials or videos that they would recommend to me that might help describe the concept? I'm ...
3
votes
1answer
142 views

Problems, which are difficult for SGD

I am doing some research on problems, for which the stochastic gradient descent doesn't perform well. Often SGD is mentioned as the best method for the training of neural networks. However, I've also ...
3
votes
0answers
41 views

How does scaled conjugate gradient work in neural network training? Comparison with gradient descent

I am very new and beginner in the machine learning world, and I would like to ask if someone could simply explain to me how does the scaled conjugate gradient method work in neural network training? ...
0
votes
0answers
57 views

Why not use line search in conjunction with stochastic gradient descent?

I'm familiar with numerical optimization in Engineering context. I have taken several graduate level engineering optimization and operations research courses. I'm beginning to learn machine learning. ...
1
vote
1answer
30 views

When can I say a network converges better?

I trained two networks with and without skip connection. The network is a FCNN and has an encoding-decoding structure. I trained the networks with SGD and MSE for 150 epochs. The attached image is a ...
1
vote
1answer
36 views

How can I improve a classification algorithm for dogs and cats?

The following code is a ML algorithm trained to classify between dogs and cats, the database is composed by 25000 images (evenly split) and can be obtained at this Link (if you click it will ...
0
votes
0answers
7 views

Difference between using SGD on new data in batches vs training on the whole dataset (recommender systems)

I work with recommender systems and use SGD to train them. I am doing both: real-time updates: updating weights as soon as a new batch of 64 entries come in and training on the whole (well part of ...
1
vote
0answers
37 views

Batch size influence on model quality

I found in https://dl.acm.org/doi/abs/10.1145/3320060 (section 3) this graph that illustrates influence of batch size. Below there is an explanation: We can show the existence of region C by ...
1
vote
1answer
25 views

Convergence under large set of learning rates

What is the interpretation of a stochastic optimization problem where a gradient descent algorithm is converging under a wide range of learning rate schedules (including ones with quite large initial ...
1
vote
1answer
176 views

What is neural network good accuracy

I am very new at machine learning, and I'm building an artificial neural network that aims to classify inputs into 2 labels. I am training the network with randomly initialized weights and through the ...
1
vote
1answer
51 views
1
vote
0answers
70 views

On projected gradient descent and inequality constraints

Consider the optimization problem \begin{equation} \min_{x\in\mathbb{R}^n} \quad f(x) \end{equation} using the gradient descent, we can iteratively solve this problem \begin{equation} x^{k+1} = x^k-\...
1
vote
0answers
21 views

Adam converges while SGD does not improve at all

I am trying to build a model based movie recommendation system with a neural network. The architecture looks as follows: ...
1
vote
0answers
72 views

Neural network doesn't converge but has good performance

I have a sequence (> 100 million) of symbols and several models predict the next symbol. To combine these predictions I'm using stacked generalization with a multilayer perceptron trained with online ...
0
votes
0answers
16 views

Image Augmentation and Online learning

I have to question. I can't find any answers online therefore I'm going to ask them here. Is Image Augmentation in the context of Object detection always meaningful? I have 100 images of a object ...
1
vote
0answers
194 views

CNN training loss regular spikes at the end of the epoch

I am training a CNN in PyTorch with Adam and the initial learning rate is 1e-5. I have 5039 samples in my epoch and the batch size is 1. I have observed that I have a regular spike pattern of training ...
0
votes
1answer
122 views

SVM: Getting number of support vectors number and relationship between C and alpha in Python sklearn SGDClassifier

I am using sklearn.SGDClassifier to train my SVM model with loss='hinge'. My questions are: Is there a way to get support vectors number by having this SGD model? I found this online but it is not ...
1
vote
1answer
195 views

How does the batch size affect the Stochastic Gradient Descent optimizer? (Example using Keras)

First of all, I know that there are lots of questions and answers about the topic throughout the site $-$ such as here, here or here (and I've probably read them all). However, I am still confused. ...
4
votes
1answer
192 views

ELBO maximization with SGD

In cases such as Gaussian mixture models, there's is no closed-term solution for the original likelihood maximization. Maximizing the ELBO, however, does have analytical update formulas (i.e. formulas ...
0
votes
1answer
226 views

Is DQN Q-Learning update really stochastic gradient descent?

In the Deepmind DQN paper, the authors mention that familiar Q Learning can be recovered by updating weights of target network at every step so if $\theta_i^-=\theta_i$ in the first loss equation, ...
1
vote
0answers
46 views

Comparing numerical stability and computing bounds on the condition number of learned weights

I have an empirical risk minimization problem with two equivalent losses that solves it, $f_1(x; \theta_1)$ and $f_2(x ; \theta_2)$, where $x$ is the data and $\theta$ are the model parameters (in ...
0
votes
0answers
10 views

High initial test validation score for Neural Network

From only the first epoch of training my NN with SGD (I use Xavier initialisation for weights), the accuracy shoots off to 92%, and then flattens out. The same thing happens with loss (but lower, of ...
1
vote
0answers
73 views

Choosing learning rate with 2nd order method - minimizing parabola in one step?

In parabola $(\theta,g)$ values are in line $(g=f'(\theta))$ - we can get slope of this line e.g. by dividing their standard deviations: $$ \mu = \frac{\sigma_\theta}{\sigma_g}=\sqrt{\frac{var(\...
1
vote
0answers
17 views

Modified Loss Function(s) for decorrelating neurons within a layer?

I'm looking for previous references on a specific topic. Does anyone know of any modified loss functions that incentivize a network to produce a diagonal neuron-to-neuron covariance matrix (averaging ...
2
votes
1answer
32 views

Why is it hard to bring back a once “broken” neural network to usable results ? - Classification MNIST SGD

I'm currently working through Michael Nielsens book "Neural Networks and Deep Learning" . If I use his code/hyperparameters the network get's quite a good classification score. Even after ...
1
vote
2answers
581 views

Why do people say gradient descent is slower than stochastic gradient descent? That's obviously not true?

With gradient descent, you calculate the gradient for the entire sample at once. With SGD, you calculate it on each sample, and then you do the same for every other sample, until you have done 1 full ...
0
votes
2answers
77 views

XOR with Neural Network [closed]

I'm trying to implement a simple neural network to fit a XOR function as shown in the book 'Deep Learning' by Ian Goodfellow, Yoshua Bengio and Aaron Courville (2016). Here is my python code using ...
4
votes
1answer
1k views

Stopping criteria for stochastic gradient descent?

When using stochastic gradient descent, how do we pick a stopping criteria? A benefit of stochastic gradient descent is that, since it is stochastic, it can avoid getting stuck in a suboptimal ...
1
vote
0answers
762 views

Time complexity of batch gradient descent

I am read http://papers.nips.cc/paper/4937-accelerating-stochastic-gradient-descent-using-predictive-variance-reduction.pdf paper. It states that "Due to the poor condition number, the standard batch ...
3
votes
1answer
78 views

The correct implementation of momentum method and NAG

Recently started a Coursera course on Deep Learning. In the optimization video, momentum and NAG were not very clearly explained so, I searched and came across the paper On the importance of ...
4
votes
0answers
66 views

Beneficial dimension for 2nd order modelling in SGD optimization?

There are currently mostly used first order methods in SGD optimizers, second order are often seen too costly as e.g. full Hessian has size $D^2$ in dimension $D$. But we don't need full Hessian - ...
1
vote
0answers
17 views

Why picking several times of same instances generally converge faster than going through instance by instance using Stochastic Gradient Descent?

I am reading Hands-on Machine Learning with Scikit-Learn & TensorFlow by Aurelien Geron. In chapter 4: Training models page 122, where it is explaining linear regression using SGD, it says that ...
1
vote
4answers
353 views

How does stochastic gradient descent even work for neural nets?

How does stochastic gradient descent (meaning where you backpropagate and adjust the weights and biases of the neural network after each single sample) even work? Doesn't that just tell the neural ...
2
votes
1answer
104 views

Can you use stochastic gradient descent with a multinomial likelihood?

I have a multinomial likelihood of the form: $$P(\underline n|\underline x) = N!\prod_{i=1}^M \frac{f_i(\underline x)^{n_i}}{n_i!}$$ where $\underline x$ is a vector of parameters, $f_i(\underline x)...
0
votes
0answers
23 views

SGD versus Adamax on XOR operator

I am trying to resolve the xor operator using neural networks, and to accomplish that this is my code: ...
2
votes
1answer
352 views

What does it mean when the global gradient norm keeps decreasing while loss has converged?

I am training an autoencoder with a $L^2$ loss. Training gives reasonable results and the loss quickly converge to a non-zero but acceptable value after about 5 epochs: However, looking at the "...
6
votes
1answer
165 views

Difference between eligibility traces and momentum?

Eligibility traces and function approximators. I'm looking at Sutton & Barto's use of eligibility traces combined with function approximation (e.g. sections 13.5, 13.6) and I noticed that it ...