Questions tagged [sgd]

Stochastic gradient descent (SGD) is a variant of gradient descent where only a small subset ("mini-batch") of training examples is used to compute the gradient on each iteration.

Filter by
Sorted by
Tagged with
3
votes
1answer
54 views

What is the probability distribution of a minibatch of data?

Suppose there are numbers $\{1, \ldots, 10\}$ You pick one at random, call it $i$ Then $i$ is a Uniform random variable (https://en.wikipedia.org/wiki/Discrete_uniform_distribution), $i \sim U\{1, 10\}...
2
votes
1answer
9 views

can we perform sub-gradient updates in mini-batches

We are already aware that in case the data is quite bulky, mini-batch gradient descent based approaches may be applied. These approaches load a mini-batch of data, compute the loss on this batch, and ...
0
votes
1answer
17 views

Anybody know of good material or videos to help me understand Stochastic Gradient Descent?

I am trying to understand stochastic gradient descent a bit better as I'm not 100. Does anybody have any materials or videos that they would recommend to me that might help describe the concept? I'm ...
3
votes
1answer
123 views

Problems, which are difficult for SGD

I am doing some research on problems, for which the stochastic gradient descent doesn't perform well. Often SGD is mentioned as the best method for the training of neural networks. However, I've also ...
2
votes
0answers
25 views

How does scaled conjugate gradient work in neural network training? Comparison with gradient descent

I am very new and beginner in the machine learning world, and I would like to ask if someone could simply explain to me how does the scaled conjugate gradient method work in neural network training? ...
0
votes
0answers
47 views

Why not use line search in conjunction with stochastic gradient descent?

I'm familiar with numerical optimization in Engineering context. I have taken several graduate level engineering optimization and operations research courses. I'm beginning to learn machine learning. ...
1
vote
1answer
28 views

When can I say a network converges better?

I trained two networks with and without skip connection. The network is a FCNN and has an encoding-decoding structure. I trained the networks with SGD and MSE for 150 epochs. The attached image is a ...
1
vote
1answer
35 views

How can I improve a classification algorithm for dogs and cats?

The following code is a ML algorithm trained to classify between dogs and cats, the database is composed by 25000 images (evenly split) and can be obtained at this Link (if you click it will ...
0
votes
0answers
7 views

Difference between using SGD on new data in batches vs training on the whole dataset (recommender systems)

I work with recommender systems and use SGD to train them. I am doing both: real-time updates: updating weights as soon as a new batch of 64 entries come in and training on the whole (well part of ...
1
vote
0answers
30 views

Batch size influence on model quality

I found in https://dl.acm.org/doi/abs/10.1145/3320060 (section 3) this graph that illustrates influence of batch size. Below there is an explanation: We can show the existence of region C by ...
1
vote
1answer
24 views

Convergence under large set of learning rates

What is the interpretation of a stochastic optimization problem where a gradient descent algorithm is converging under a wide range of learning rate schedules (including ones with quite large initial ...
1
vote
1answer
55 views

What is neural network good accuracy

I am very new at machine learning, and I'm building an artificial neural network that aims to classify inputs into 2 labels. I am training the network with randomly initialized weights and through the ...
0
votes
1answer
42 views
1
vote
0answers
57 views

On projected gradient descent and inequality constraints

Consider the optimization problem \begin{equation} \min_{x\in\mathbb{R}^n} \quad f(x) \end{equation} using the gradient descent, we can iteratively solve this problem \begin{equation} x^{k+1} = x^k-\...
1
vote
0answers
19 views

Adam converges while SGD does not improve at all

I am trying to build a model based movie recommendation system with a neural network. The architecture looks as follows: ...
1
vote
0answers
72 views

Neural network doesn't converge but has good performance

I have a sequence (> 100 million) of symbols and several models predict the next symbol. To combine these predictions I'm using stacked generalization with a multilayer perceptron trained with online ...
0
votes
0answers
14 views

Image Augmentation and Online learning

I have to question. I can't find any answers online therefore I'm going to ask them here. Is Image Augmentation in the context of Object detection always meaningful? I have 100 images of a object ...
1
vote
0answers
120 views

CNN training loss regular spikes at the end of the epoch

I am training a CNN in PyTorch with Adam and the initial learning rate is 1e-5. I have 5039 samples in my epoch and the batch size is 1. I have observed that I have a regular spike pattern of training ...
0
votes
1answer
70 views

SVM: Getting number of support vectors number and relationship between C and alpha in Python sklearn SGDClassifier

I am using sklearn.SGDClassifier to train my SVM model with loss='hinge'. My questions are: Is there a way to get support vectors number by having this SGD model? I found this online but it is not ...
0
votes
0answers
11 views

Convergence analysis of fine tuning or transfer learning

Fine-tuning is the process of using a pre-trained model (corresponding to an old dataset) for learning a new task (for a new dataset). I have looked a lot but could not find a convergence analysis for ...
0
votes
0answers
61 views

Moments accountant beyond subsampled Gaussian mechanism

Moments accountant has been in the first place applied on the subsampled Gaussian mechanism, leading to tight privacy cost estimation and efficient differentially private SGD-based learning in neural ...
1
vote
1answer
112 views

How does the batch size affect the Stochastic Gradient Descent optimizer? (Example using Keras)

First of all, I know that there are lots of questions and answers about the topic throughout the site $-$ such as here, here or here (and I've probably read them all). However, I am still confused. ...
4
votes
1answer
165 views

ELBO maximization with SGD

In cases such as Gaussian mixture models, there's is no closed-term solution for the original likelihood maximization. Maximizing the ELBO, however, does have analytical update formulas (i.e. formulas ...
0
votes
1answer
149 views

Is DQN Q-Learning update really stochastic gradient descent?

In the Deepmind DQN paper, the authors mention that familiar Q Learning can be recovered by updating weights of target network at every step so if $\theta_i^-=\theta_i$ in the first loss equation, ...
1
vote
0answers
34 views

Comparing numerical stability and computing bounds on the condition number of learned weights

I have an empirical risk minimization problem with two equivalent losses that solves it, $f_1(x; \theta_1)$ and $f_2(x ; \theta_2)$, where $x$ is the data and $\theta$ are the model parameters (in ...
0
votes
0answers
10 views

High initial test validation score for Neural Network

From only the first epoch of training my NN with SGD (I use Xavier initialisation for weights), the accuracy shoots off to 92%, and then flattens out. The same thing happens with loss (but lower, of ...
1
vote
0answers
64 views

Choosing learning rate with 2nd order method - minimizing parabola in one step?

In parabola $(\theta,g)$ values are in line $(g=f'(\theta))$ - we can get slope of this line e.g. by dividing their standard deviations: $$ \mu = \frac{\sigma_\theta}{\sigma_g}=\sqrt{\frac{var(\...
1
vote
0answers
14 views

Modified Loss Function(s) for decorrelating neurons within a layer?

I'm looking for previous references on a specific topic. Does anyone know of any modified loss functions that incentivize a network to produce a diagonal neuron-to-neuron covariance matrix (averaging ...
2
votes
1answer
29 views

Why is it hard to bring back a once “broken” neural network to usable results ? - Classification MNIST SGD

I'm currently working through Michael Nielsens book "Neural Networks and Deep Learning" . If I use his code/hyperparameters the network get's quite a good classification score. Even after ...
0
votes
0answers
26 views

Stochastic/batch gradient descent (type SGD/ADAM) with weighted mean square error loss

Assume I assign uneven weights to losses of different examples, i.e. I set my SGD/ADAM to train a universal approximator f (e.g. a neural net) by minimization of a weighted mean square error: $ L = \...
0
votes
0answers
17 views

For large $n$, what tuning parameters need to be changed in SGD?

In stochastic descent algorithms, what happens if we increase the data size that we are working with? Does this cause a change to the tuning parameters that we can use? For example, say you are ...
0
votes
2answers
310 views

Why do people say gradient descent is slower than stochastic gradient descent? That's obviously not true?

With gradient descent, you calculate the gradient for the entire sample at once. With SGD, you calculate it on each sample, and then you do the same for every other sample, until you have done 1 full ...
0
votes
2answers
55 views

XOR with Neural Network [closed]

I'm trying to implement a simple neural network to fit a XOR function as shown in the book 'Deep Learning' by Ian Goodfellow, Yoshua Bengio and Aaron Courville (2016). Here is my python code using ...
4
votes
1answer
847 views

Stopping criteria for stochastic gradient descent?

When using stochastic gradient descent, how do we pick a stopping criteria? A benefit of stochastic gradient descent is that, since it is stochastic, it can avoid getting stuck in a suboptimal ...
1
vote
0answers
578 views

Time complexity of batch gradient descent

I am read http://papers.nips.cc/paper/4937-accelerating-stochastic-gradient-descent-using-predictive-variance-reduction.pdf paper. It states that "Due to the poor condition number, the standard batch ...
3
votes
1answer
61 views

The correct implementation of momentum method and NAG

Recently started a Coursera course on Deep Learning. In the optimization video, momentum and NAG were not very clearly explained so, I searched and came across the paper On the importance of ...
4
votes
0answers
60 views

Beneficial dimension for 2nd order modelling in SGD optimization?

There are currently mostly used first order methods in SGD optimizers, second order are often seen too costly as e.g. full Hessian has size $D^2$ in dimension $D$. But we don't need full Hessian - ...
1
vote
0answers
17 views

Why picking several times of same instances generally converge faster than going through instance by instance using Stochastic Gradient Descent?

I am reading Hands-on Machine Learning with Scikit-Learn & TensorFlow by Aurelien Geron. In chapter 4: Training models page 122, where it is explaining linear regression using SGD, it says that ...
1
vote
4answers
315 views

How does stochastic gradient descent even work for neural nets?

How does stochastic gradient descent (meaning where you backpropagate and adjust the weights and biases of the neural network after each single sample) even work? Doesn't that just tell the neural ...
2
votes
1answer
99 views

Can you use stochastic gradient descent with a multinomial likelihood?

I have a multinomial likelihood of the form: $$P(\underline n|\underline x) = N!\prod_{i=1}^M \frac{f_i(\underline x)^{n_i}}{n_i!}$$ where $\underline x$ is a vector of parameters, $f_i(\underline x)...
0
votes
0answers
19 views

SGD versus Adamax on XOR operator

I am trying to resolve the xor operator using neural networks, and to accomplish that this is my code: ...
2
votes
1answer
240 views

What does it mean when the global gradient norm keeps decreasing while loss has converged?

I am training an autoencoder with a $L^2$ loss. Training gives reasonable results and the loss quickly converge to a non-zero but acceptable value after about 5 epochs: However, looking at the "...
5
votes
1answer
144 views

Difference between eligibility traces and momentum?

Eligibility traces and function approximators. I'm looking at Sutton & Barto's use of eligibility traces combined with function approximation (e.g. sections 13.5, 13.6) and I noticed that it ...
1
vote
1answer
903 views

Combining Random forest with Adam (or an other gradient method)

There is no "gradient" in the standard Random Forest formulation, but can I combine random Forests with an optimisation method like Gradient Descent or SGD? Can I use Adam (Adaptive moment estimation)...
7
votes
1answer
1k views

Does Keras SGD optimizer implement batch, mini-batch, or stochastic gradient descent?

I am a newbie in Deep Learning libraries and thus decided to go with Keras. While implementing a NN model, I saw the batch_size parameter in ...
6
votes
1answer
970 views

Can small SGD batch size lead to faster overfitting?

I have feedforward neural net, trained on cca 34k samples and tested on 8k samples. There is 139 features in dataset. The ANN does classification between two labels, 0 and 1, so I am using sigmoid ...
1
vote
0answers
65 views

Momentum updates average of $g$, Adagrad also of $g^2$ - any other interesting updated averages for SGD convergence?

Updating exponential moving average is a basic tool of SGD methods, starting with of gradient $g$ in momentum method to extract local linear trend from the statistics. Then e.g. Adagrad, ADAM family ...
1
vote
1answer
232 views

Saddle-free Newton method for SGD - while Newton attracts saddles, is it worth to actively replel them?

While 2nd order methods have many advantages, e.g. natural gradient (e.g. in L-BFGS) attracts to close zero gradient point, which is usually saddle. Other try to pretend that our very non-convex ...
1
vote
0answers
42 views

What are good packages for online linear regression besides Vowpal Wabbit?

Does anyone know of online learning packages that implement NG and NAG algorithms from Stephen Ross' paper: chrome-extension://oemmndcbldboiebfnladdacbdfmadadm/http://auai.org/uai2013/prints/papers/...
4
votes
1answer
352 views

Is stochastic gradient descent pseudo-stochastic?

I know that stochastic gradient descent randomly chooses 1 sample to update the weights. An epoch is defined as using all $N$ samples. So with SGD, for each epoch, we update the weights $N$ times. ...