Questions tagged [sgd]

Stochastic gradient descent (SGD) is a variant of gradient descent where only a small subset ("mini-batch") of training examples is used to compute the gradient on each iteration.

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

Convergence under large set of learning rates

What is the interpretation of a stochastic optimization problem where a gradient descent algorithm is converging under a wide range of learning rate schedules (including ones with quite large initial ...
1
vote
1answer
19 views

What is neural network good accuracy

I am very new at machine learning, and I'm building an artificial neural network that aims to classify inputs into 2 labels. I am training the network with randomly initialized weights and through the ...
0
votes
1answer
32 views
1
vote
0answers
36 views

On projected gradient descent and inequality constraints

Consider the optimization problem \begin{equation} \min_{x\in\mathbb{R}^n} \quad f(x) \end{equation} using the gradient descent, we can iteratively solve this problem \begin{equation} x^{k+1} = x^k-\...
0
votes
0answers
25 views

Is there a way to figure out using filter sizes (manually) how many operations Batchnorm, Conv, and Relu layers take during backprop? [duplicate]

I'm working with a basic resnet model. I want to understand how to compute by-hand the number of ops for a specific layer during backward pass (backprop) during Training (not inference). This involves ...
1
vote
0answers
13 views

Adam converges while SGD does not improve at all

I am trying to build a model based movie recommendation system with a neural network. The architecture looks as follows: ...
1
vote
0answers
72 views

Neural network doesn't converge but has good performance

I have a sequence (> 100 million) of symbols and several models predict the next symbol. To combine these predictions I'm using stacked generalization with a multilayer perceptron trained with online ...
0
votes
0answers
12 views

Image Augmentation and Online learning

I have to question. I can't find any answers online therefore I'm going to ask them here. Is Image Augmentation in the context of Object detection always meaningful? I have 100 images of a object ...
1
vote
0answers
60 views

CNN training loss regular spikes at the end of the epoch

I am training a CNN in PyTorch with Adam and the initial learning rate is 1e-5. I have 5039 samples in my epoch and the batch size is 1. I have observed that I have a regular spike pattern of training ...
0
votes
1answer
40 views

SVM: Getting number of support vectors number and relationship between C and alpha in Python sklearn SGDClassifier

I am using sklearn.SGDClassifier to train my SVM model with loss='hinge'. My questions are: Is there a way to get support vectors number by having this SGD model? I found this online but it is not ...
0
votes
0answers
6 views

Convergence analysis of fine tuning or transfer learning

Fine-tuning is the process of using a pre-trained model (corresponding to an old dataset) for learning a new task (for a new dataset). I have looked a lot but could not find a convergence analysis for ...
0
votes
0answers
32 views

Moments accountant beyond subsampled Gaussian mechanism

Moments accountant has been in the first place applied on the subsampled Gaussian mechanism, leading to tight privacy cost estimation and efficient differentially private SGD-based learning in neural ...
1
vote
1answer
68 views

How does the batch size affect the Stochastic Gradient Descent optimizer? (Example using Keras)

First of all, I know that there are lots of questions and answers about the topic throughout the site $-$ such as here, here or here (and I've probably read them all). However, I am still confused. ...
4
votes
1answer
151 views

ELBO maximization with SGD

In cases such as Gaussian mixture models, there's is no closed-term solution for the original likelihood maximization. Maximizing the ELBO, however, does have analytical update formulas (i.e. formulas ...
0
votes
1answer
94 views

Is DQN Q-Learning update really stochastic gradient descent?

In the Deepmind DQN paper, the authors mention that familiar Q Learning can be recovered by updating weights of target network at every step so if $\theta_i^-=\theta_i$ in the first loss equation, ...
0
votes
0answers
29 views

Comparing numerical stability and computing bounds on the condition number of learned weights

I have an empirical risk minimization problem with two equivalent losses that solves it, $f_1(x; \theta_1)$ and $f_2(x ; \theta_2)$, where $x$ is the data and $\theta$ are the model parameters (in ...
0
votes
0answers
10 views

High initial test validation score for Neural Network

From only the first epoch of training my NN with SGD (I use Xavier initialisation for weights), the accuracy shoots off to 92%, and then flattens out. The same thing happens with loss (but lower, of ...
1
vote
0answers
59 views

Choosing learning rate with 2nd order method - minimizing parabola in one step?

In parabola $(\theta,g)$ values are in line $(g=f'(\theta))$ - we can get slope of this line e.g. by dividing their standard deviations: $$ \mu = \frac{\sigma_\theta}{\sigma_g}=\sqrt{\frac{var(\...
1
vote
0answers
13 views

Modified Loss Function(s) for decorrelating neurons within a layer?

I'm looking for previous references on a specific topic. Does anyone know of any modified loss functions that incentivize a network to produce a diagonal neuron-to-neuron covariance matrix (averaging ...
2
votes
1answer
28 views

Why is it hard to bring back a once “broken” neural network to usable results ? - Classification MNIST SGD

I'm currently working through Michael Nielsens book "Neural Networks and Deep Learning" . If I use his code/hyperparameters the network get's quite a good classification score. Even after the first ...
0
votes
0answers
24 views

Stochastic/batch gradient descent (type SGD/ADAM) with weighted mean square error loss

Assume I assign uneven weights to losses of different examples, i.e. I set my SGD/ADAM to train a universal approximator f (e.g. a neural net) by minimization of a weighted mean square error: $ L = \...
0
votes
0answers
17 views

For large $n$, what tuning parameters need to be changed in SGD?

In stochastic descent algorithms, what happens if we increase the data size that we are working with? Does this cause a change to the tuning parameters that we can use? For example, say you are ...
0
votes
2answers
125 views

Why do people say gradient descent is slower than stochastic gradient descent? That's obviously not true?

With gradient descent, you calculate the gradient for the entire sample at once. With SGD, you calculate it on each sample, and then you do the same for every other sample, until you have done 1 full ...
0
votes
2answers
48 views

XOR with Neural Network [closed]

I'm trying to implement a simple neural network to fit a XOR function as shown in the book 'Deep Learning' by Ian Goodfellow, Yoshua Bengio and Aaron Courville (2016). Here is my python code using ...
3
votes
1answer
470 views

Stopping criteria for stochastic gradient descent?

When using stochastic gradient descent, how do we pick a stopping criteria? A benefit of stochastic gradient descent is that, since it is stochastic, it can avoid getting stuck in a suboptimal ...
0
votes
0answers
77 views

Bayesian interpretation of gradient clipping

In the context of Bayesian interpretations of SGD for neural network training, is there an interpretation for the gradient clipping operation which is often included?
1
vote
0answers
419 views

Time complexity of batch gradient descent

I am read http://papers.nips.cc/paper/4937-accelerating-stochastic-gradient-descent-using-predictive-variance-reduction.pdf paper. It states that "Due to the poor condition number, the standard batch ...
3
votes
1answer
50 views

The correct implementation of momentum method and NAG

Recently started a Coursera course on Deep Learning. In the optimization video, momentum and NAG were not very clearly explained so, I searched and came across the paper On the importance of ...
3
votes
0answers
55 views

Beneficial dimension for 2nd order modelling in SGD optimization?

There are currently mostly used first order methods in SGD optimizers, second order are often seen too costly as e.g. full Hessian has size $D^2$ in dimension $D$. But we don't need full Hessian - ...
0
votes
0answers
22 views

Why LMS filter can be regarded as SGD?

My question is about LMS algorithm. Why say LMS filter is Stochastic Gradient Descent? Can I apply SVRG (Stochastic Variance reduced gradient) for LMS filter? And...What's wrong with this code? <...
1
vote
0answers
17 views

Why picking several times of same instances generally converge faster than going through instance by instance using Stochastic Gradient Descent?

I am reading Hands-on Machine Learning with Scikit-Learn & TensorFlow by Aurelien Geron. In chapter 4: Training models page 122, where it is explaining linear regression using SGD, it says that ...
1
vote
4answers
290 views

How does stochastic gradient descent even work for neural nets?

How does stochastic gradient descent (meaning where you backpropagate and adjust the weights and biases of the neural network after each single sample) even work? Doesn't that just tell the neural ...
2
votes
1answer
97 views

Can you use stochastic gradient descent with a multinomial likelihood?

I have a multinomial likelihood of the form: $$P(\underline n|\underline x) = N!\prod_{i=1}^M \frac{f_i(\underline x)^{n_i}}{n_i!}$$ where $\underline x$ is a vector of parameters, $f_i(\underline x)...
0
votes
0answers
35 views

What's the bug in my implementation/understanding of backpropagation?

For learning purposes, I'm trying to implement a simple neural network with only linear layers followed by logistic activation. As far as I understand, the backpropagation algorithm exploits the ...
0
votes
0answers
150 views

Variance when using mini-batch stochastic gradient descent

When using mini-batch stochastic gradient descent, we have the option of either summing the observation specific gradient estimates, or we can take an average over the observation-specific gradient ...
0
votes
0answers
17 views

SGD versus Adamax on XOR operator

I am trying to resolve the xor operator using neural networks, and to accomplish that this is my code: ...
2
votes
1answer
140 views

What does it mean when the global gradient norm keeps decreasing while loss has converged?

I am training an autoencoder with a $L^2$ loss. Training gives reasonable results and the loss quickly converge to a non-zero but acceptable value after about 5 epochs: However, looking at the "...
3
votes
1answer
112 views

Difference between eligibility traces and momentum?

Eligibility traces and function approximators. I'm looking at Sutton & Barto's use of eligibility traces combined with function approximation (e.g. sections 13.5, 13.6) and I noticed that it ...
1
vote
1answer
768 views

Combining Random forest with Adam (or an other gradient method)

There is no "gradient" in the standard Random Forest formulation, but can I combine random Forests with an optimisation method like Gradient Descent or SGD? Can I use Adam (Adaptive moment estimation)...
4
votes
1answer
843 views

Does Keras SGD optimizer implement batch, mini-batch, or stochastic gradient descent?

I am a newbie in Deep Learning libraries and thus decided to go with Keras. While implementing a NN model, I saw the batch_size parameter in ...
6
votes
1answer
690 views

Can small SGD batch size lead to faster overfitting?

I have feedforward neural net, trained on cca 34k samples and tested on 8k samples. There is 139 features in dataset. The ANN does classification between two labels, 0 and 1, so I am using sigmoid ...
1
vote
0answers
48 views

Momentum updates average of g, Adagrad also of g^2 - any other interesting updated averages for SGD convergence?

Updating exponential moving average is a basic tool of SGD methods, starting with of gradient $g$ in momentum method to extract local linear trend from the statistics. Then e.g. Adagrad, ADAM family ...
1
vote
0answers
135 views

Saddle-free Newton method for SGD - while Newton attracts saddles, is it worth to actively replel them?

While 2nd order methods have many advantages, e.g. natural gradient (e.g. in L-BFGS) attracts to close zero gradient point, which is usually saddle. Other try to pretend that our very non-convex ...
1
vote
0answers
39 views

What are good packages for online linear regression besides Vowpal Wabbit?

Does anyone know of online learning packages that implement NG and NAG algorithms from Stephen Ross' paper: chrome-extension://oemmndcbldboiebfnladdacbdfmadadm/http://auai.org/uai2013/prints/papers/...
4
votes
1answer
340 views

Is stochastic gradient descent pseudo-stochastic?

I know that stochastic gradient descent randomly chooses 1 sample to update the weights. An epoch is defined as using all $N$ samples. So with SGD, for each epoch, we update the weights $N$ times. ...
1
vote
0answers
123 views

How positive definite Hessian approximations for SGD (e.g. Gauss-Newton) handle saddles?

For example due to symmetry of parameters, functions optimized in machine learning usually have huge number of local minima and saddles - growing exponentially with dimension. I am trying to ...
6
votes
1answer
324 views

Do there exist adaptive step size methods for Newton-Raphson optimization?

Stochastic/Mini-batch gradient descent, caused by interest in deep learning, has made lots of advances in adaptive step sizes. For example, Adam, Nadam, Adamax, ..., are all improvements to the ...
12
votes
1answer
2k views

Why second order SGD convergence methods are unpopular for deep learning?

It seems that, especially for deep learning, there are dominating very simple methods for optimizing SGD convergence like ADAM - nice overview: http://ruder.io/optimizing-gradient-descent/ They trace ...
0
votes
1answer
161 views

Is training loss guaranteed to decrease for stochastic gradient descent? [duplicate]

When performing stochastic gradient descent, it is necessary for the training loss to decrease a) between iterations in an epoch? (I think the answer is no) b) between epochs? (I think the answer is ...
5
votes
0answers
56 views

Why divide the learning rate by the size of the mini batch? [duplicate]

In Michael Nielsen's online book Neural Networks and Deep Learning, in chapter one (and onwards) he divides the learning rate, $\eta$, by the size of the mini batch when he performs stochastic ...