Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
Make your voice heard. Take the 2019 Developer Survey now

Questions tagged [sgd]

Stochastic gradient descent (SGD) is a variant of gradient descent where only a small subset ("mini-batch") of training examples is used to compute the gradient on each iteration.

0
votes
0answers
14 views

Accelerated Projected SGD under box constraints

Are there generalizations of ADAM or Adagrad algorithm that allow box constraints for the parameters to be incorporated in the gradient descent step? Is it valid to simply run the algorithm as usual ...
0
votes
1answer
39 views

When does my unsupervised autoencoder start to overfit?

I am working on anomaly detection using an autoencoder neural network with $1$ hidden layer. This is an unsupervised setting, as I do not have previous examples of anomalies. The input data has ...
0
votes
0answers
42 views

Natural Gradients in Stochastic Variational Inference (SVI) for Gaussian Process Regression

Currently, I've hard times in understanding the natural gradients update in SVI method for Gaussian Process. I'm learning the SVI method for Gaussian Process through Gaussian Process for Big Data ...
2
votes
1answer
41 views

Why increasing the batch size has the same effect as decaying the learning rate?

There have been a few papers this year, concerned with very large scale training, where instead than decaying the learning rate $\eta$, the batch size $B$ was increased, usually with the same schedule ...
1
vote
1answer
26 views

How to understand whether Stochastic Gradient Descent has converged?

I am using SGD to solve for MSE function. My training set is around 50K, and I am monitoring the gradient at every epoch (once a pass is completed over all the training data). I played around a lot ...
0
votes
0answers
13 views

How to choose the learning rate for stochastic gradient descent (via backtracking)?

I am trying to implement "from scratch" SGD and Mini Batch Gradient Descent in Matlab. I have to minimize a function like $f(x)= \sum f_i (X_i, y_i)$ where $(X_i, y_i)$ is a data point (features and ...
0
votes
2answers
34 views

Is downsampling okay for logistic regression if I only care about relative ordering (ROC AUC)?

I see a few discussions that suggest downsampling is never correct for logistic regression or suggesting that you have to do bias term corrections post-hoc: Downsampling vs upsampling on the ...
0
votes
0answers
30 views

need help understanding the benefit of score function estimator

The score function estimator a.k.a REINFORCE policy gradient in reinforcement learning is (from http://blog.shakirm.com/2015/11/machine-learning-trick-of-the-day-5-log-derivative-trick/): \begin{...
1
vote
0answers
151 views

Stochastic gradient descent vs mini-batch gradient descent

Gradient descent in neural networks involves the whole dataset for each weights-update step, and it is well known it would be computationally too long and also could make it converge to a local non-...
0
votes
1answer
41 views

SGD and quantile regression

It is my understanding that the quantile loss is not differentiable (at 0) so base gradient descent cannot be used. However, Vowpal Wabbit which is an SGD-based learner very much includes quantile ...
0
votes
0answers
26 views

Is it possible to combine SPSA and Adam?

In SGD algorithms such as Adam you generally make a bad estimate of the gradient of the loss function and take that gradient to move the parameters in the desired direction. Gradient free methods ...
1
vote
0answers
15 views

Stochastic gradient descent (SGD) on data with weights

Mostly deep learning model training is on data with a unit weight. In this case, every mini-batch of a fixed size, say, 32, contains exactly the same total weight (32) for each update. This is the ...
1
vote
0answers
38 views

Difference between Stochastic Gradient Descent and Sklearn's Stochastic Average Gradient (SAG) solver?

How does stochastic gradient descent varies from Sklearn's SAG (Stochastic average gradient) solver? Edit: Many sklearn models like Ridge, LogisticRegression, etc accept SAG as a solver
0
votes
1answer
195 views

stochastic gradient descent of ridge regression when regularization parameter is very big

As we know, the gradient of ridge regression is: $$ g = \frac{\partial L}{\partial \theta} = -X_i^T(y_i-X_i\theta)+2\lambda\theta $$ where $X_i$ is the $i$th training sample. The update of $\theta$ is ...
23
votes
6answers
3k views

For convex problems, does gradient in Stochastic Gradient Descent (SGD) always point at the global extreme value?

Given a convex cost function, using SGD for optimization, we will have a gradient (vector) at a certain point during the optimization process. My question is, given the point on the convex, does the ...
1
vote
1answer
112 views

Why doesn't feature standardization make SGD with momentum redundant?

In the paper An overview of gradient descent optimization algorithms, the author discusses the Momentum algorithm: SGD has trouble navigating ravines, i.e. areas where the surface curves much ...
-1
votes
1answer
40 views

Stochastic gradient descent and asymptotic analysis

In 8th chapter of deep learning book, the following lines are written under Stochastic gradient descent heading: The asymptotic analysis obscures many advantages that stochastic gradient descent ...
3
votes
1answer
216 views

Is stochastic gradient descent biased?

In the paper Mutual Information Neural Estimation, the authors derive the following gradient for the network $$ \nabla_\theta\mathcal V(\theta)=\mathbb E\left[\nabla_\theta T_\theta\right]-{\mathbb E\...
0
votes
0answers
22 views

How to deal with numeric instability in stochastic gradient descent?

Imagine that we try to perform sgd using a gradient that takes very small or very large values (e.g. it is a product of many terms that are larger than 1). Is there a standard approach to deal with ...
0
votes
2answers
39 views

Is a loss function computed after each step of gradient descent or after a whole epoch?

In neural networks with mini-batch or stochastic gradient descent, is a loss function computed after each step of gradient descent or after a whole epoch?
1
vote
0answers
38 views

Support Vector Machines: a beginner's question about the underlying math

I'm new to Support Vector Machines and I've been trying to get into the underlying math (instead of just using Scikit Learn or something like that). I understand the math behind it up to the point ...
1
vote
1answer
27 views

Regularization and weight updates in SGD

I am using single observation to compute losses using neural network implementation in Pytorch. I am confused in a small detail of SGD. If I compute loss and do loss.backward(), I am accumulating ...
1
vote
0answers
114 views

Simulated Annealing vs SGD with (warm) Restarts

What's the difference between simulated annealing and stochastic gradient descent with restarts? They both seem like they are occasionally going backwards at a decreasing rate. Also what is the ...
0
votes
0answers
42 views

Is the regularization term necessary when classifying one feature?

I'm using the Stochastic Gradient Descent linear classifier (implemented in Scikit-learn) to classify an image pixel by pixel. So my dataset has only one feature, ...
0
votes
1answer
30 views

Stochastic gradient descent update

Equation 93 of Chapter 3 of Michael Nielsen's neural networks book describes the stochastic gradient descent update rule as the following: $w \leftarrow (1-\frac{\eta\lambda}{n})w - \frac{\eta}{m}\...
3
votes
1answer
236 views

Why is gradient descent with momentum considered an exponentially weighted average?

I recently watched Andrew Ng's video on SGDM. I understand that the momentum term updates the gradient by weighting the last gradient and using a small component of V_dw. I don't understand why ...
0
votes
1answer
28 views

How to define nearest neighbor search such that it can be optimized using stochastic gradient descent?

Assume that there is a reference two-dimensional array ref and a given vector x. I would like to return the closest vector to <...
6
votes
1answer
227 views

What is the difference between VAE and Stochastic Backpropagation for Deep Generative Models?

What is the difference between Auto-encoding Variational Bayes and Stochastic Backpropagation for Deep Generative Models? Does inference in both methods lead to the same results? I'm not aware of any ...
1
vote
0answers
34 views

Distribution of coefficients arrived at by stochastic gradient descent?

In SGD we have update $$w^{(k)} = w^{(k-1)} + \nabla _wL(y,w)$$ Hence $w^{(k)}$ is a sum of random variables. They're not iid, so the central limit theorem doesn't apply. Is there some result, ...
1
vote
2answers
97 views

How to set up a linear system to interpolate the train data perfectly with Gradient Descent?

Consider a (consistent) regression problem (i.e. we are trying to predict a real valued function and we don't have inconsistencies in the way we map x's to y). I am trying to perfectly fit/...
3
votes
1answer
95 views

Which method for training neural networks works best?

There are a dozens of different algorithms for training NNs. Most of them are stochastic gradient descent with some variations. Is there a comparative study that demonstrates some of them to be ...
1
vote
1answer
71 views

Unbiased estimation of convex function of a sum of random variables

Let $X_1, \ldots, X_N $ be $N$ fixed numbers. If we want an unbiased estimate of $M = (X_1 + \ldots + X_N)/N$ without actually doing an $O(n)$ sum then we can just sample a point uniformly at random, ...
2
votes
2answers
733 views

Stochastic gradient descent Vs Mini-batch size 1

Is stochastic gradient descent basically the name given to mini-batch training where batch size = 1 and selecting random training rows? i.e. it is the same as 'normal' gradient descent, it's just the ...
-1
votes
1answer
50 views

Avoiding Matlab's underflow prevention in backprop, due to performance cost

In Matlab, I understand that if a number gets closer to zero than realmin, then Matlab converts the double to a denorm . I am noticing this causes significant ...
0
votes
0answers
52 views

ADAM Gradient descent oscillates close to minimum

I am using ADAM as an optimization algorithm to minimize some black box function $f(x,y)$. I know this function is convex and has a minimum $f(5,5) = 0$. Initially, the algorithm proceeds as expected:...
3
votes
1answer
234 views

Why no one talks about stochastic conjugate gradient descent?

As is known to all, stochastic gradient descent is a popular optimizer in machine learning. There have been many variants of SGD. However, it has come to my attention that no one talks about the ...
1
vote
0answers
210 views

Hinton claims SGD with batch norm can help: How?

In Hinton's paper "Layer Normalization", on the first page he says Feedforward neural networks trained using batch normalization converge faster even with simple SGD. By this I think he means ...
1
vote
1answer
169 views

Should I use the whole dataset in the forward pass when doing minibatch gradient descent?

I've implemented the following algorithm. For each minibatch: Compute the gradient using the mini-batch sample Update the parameters Update the hidden layers. If $\Gamma_L$ are the new parameters ...
2
votes
1answer
609 views

Stochastic Gradient Descent, Mini-Batch and Batch Gradient Descent

I was learning the optimization part in deep learning. Let's take linear regression as a simple example. Let $m$ be the total number of data points in the training set $(X,y)$ and $n$ is the number ...
1
vote
0answers
57 views

Is mini-batch / stochastic gradient descend similar implicitly adding the same effect as simulated annealing?

Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak Peter Tang. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. https://arxiv.org/...
1
vote
1answer
468 views

The actual role of second-order optimization as oppose to first-order optimizations

I do not fully understand how second-order optimization approaches help machine learning algorithms, like multilayer perceptron, to achieve the global minimum error....
4
votes
0answers
85 views

Equivalent Gradients in Kernelized SVM

Let $\varphi: \mathcal{X} \to \mathcal{H}$ a mapping with corresponding kernel $K:\mathcal{X}\times\mathcal{X}\to \mathbb{R}$ (that is, $K\left(x,x'\right) = \left<\varphi\left(x\right), \varphi\...
7
votes
2answers
1k views

Gradient descent on non-convex functions

What situations do we know of where gradient descent can be shown to converge (either to a critical point or to a local/global minima) for non-convex functions? For SGD on non-convex functions, one ...
0
votes
1answer
296 views

Difference between stochastic variational inference and variational inference?

Very simple, as the question header says: what is the difference between SVI and VI? I cannot seem to find a clear-cut definition online.
2
votes
1answer
76 views

Matrix factorization for expanding matrix

In the paper Matrix Factorization Techniques for Recommender Systems Koren, Bell and Volinsky describe how the matrix $R_{n \times k}$ (users $\times$ movie ratings) can be decomposed to $P_{n \times ...
0
votes
0answers
59 views

Can SGDClassifer be used in python as a similar algorithm as glmnet in R?

I m doing a fraud detection (bank) analysis, where in the objective is to detect actual frauds as closely as possible. Hence, Sensitivity needs to be maximum. I have a learning_data and an ...
3
votes
2answers
784 views

Gradient Bandit Algorithm baseline

I am reading Sutton's latest draft of "Reinforcement learning, an introduction" and I came to the Gradient Bandit Algorithm (page 29). I am having a bit of trouble understanding how the baseline ...
2
votes
0answers
148 views

Difference between Stochastic Approximation (SA) and Stochastic Gradient Descent (SGD)

I understand the intended use cases for both stochastic approximation algorithms like SPSA or FDSA, and for SGD algorithms like Adam. SPSA is intended for noisy objective functions, and Adam for ...
2
votes
1answer
261 views

What should we do when changing SGD optimizer to Adam optimizer?

Adam is one popular method of the optimization policies with adaptive learning rate. I'm focusing on a image segmentation project using fully convolutional networks. All weights were initialized by ...
1
vote
0answers
80 views

Behavior of AdaGrad without the square root in the denominator

Multiple articles claim that AdaGrad does not work well when the square-root in the formula is not taken. This is one such example. $\theta_{t+1,i} = \theta_{t,i}-\dfrac{\eta}{\sqrt{G_{t,ii}+\epsilon}...