Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [sgd]

Stochastic gradient descent (SGD) is a variant of gradient descent where only a small subset ("mini-batch") of training examples is used to compute the gradient on each iteration.

0
votes
0answers
19 views

Is Stochastic Gradient Descent sensitive to training permutation?

I've recently read that SGD (Stochastic Gradient Descent) is one of the most popular techniques for training Machine Learning algorithms, including DNNs (deep neural networks). However, my ...
0
votes
0answers
12 views

What is the relationship between stochastic mirror descent and stochastic gradient descent?

I don't know much about stochastic mirror descent and was wondering if someone could briefly summarize it in general terms and compare/contrast it to stochastic gradient descent. When I understand ...
0
votes
0answers
25 views

Difference between eligibility traces and momentum?

Eligibility traces and function approximators. I'm looking at Sutton & Barto's use of eligibility traces combined with function approximation (e.g. sections 13.5, 13.6) and I noticed that it ...
0
votes
1answer
79 views

Combining Random forest with Adam (or an other gradient method)

There is no "gradient" in the standard Random Forest formulation, but can I combine random Forests with an optimisation method like Gradient Descent or SGD? Can I use Adam (Adaptive moment estimation)...
0
votes
0answers
24 views

Minimizing expected loss with non-fixed probability distribution

Is there any convergence studies or algorithm to solve the following problem? $$ \mathbf{\hat{w}} = \min_\mathbf{w} \int\mathcal{L}(\mathbf{x};\mathbf{w})P(\mathbf{x};\mathbf{w})\ \mathrm{d}\mathbf{x}...
2
votes
1answer
25 views

Does Keras SGD optimizer implement batch, mini-batch, or stochastic gradient descent?

I am a newbie in Deep Learning libraries and thus decided to go with Keras. While implementing a NN model, I saw the batch_size parameter in ...
0
votes
0answers
6 views

Stochastic Gradient Descent: using minibatch AND small dataset size

My dataset has textual samples where the number of the samples is <1000. I'm training an NERecogniser by using spaCy library. Is using minibatch important in my case (small dataset size)? I ...
5
votes
1answer
69 views

Can small SGD batch size lead to faster overfitting?

I have feedforward neural net, trained on cca 34k samples and tested on 8k samples. There is 139 features in dataset. The ANN does classification between two labels, 0 and 1, so I am using sigmoid ...
0
votes
0answers
21 views

Singular values for a latent-factor model

Suppose we build a latent-factor model using alternating least squares (ALS) or stochastic gradient descent (SGD). Can we calculate weights for each latent factor, in a similar way to how the singular-...
1
vote
0answers
25 views

Momentum updates average of g, Adagrad also of g^2 - any other interesting updated averages for SGD convergence?

Updating exponential moving average is a basic tool of SGD methods, starting with of gradient $g$ in momentum method to extract local linear trend from the statistics. Then e.g. Adagrad, ADAM family ...
1
vote
0answers
26 views

Saddle-free Newton method for SGD - while Newton attracts saddles, is it worth to actively replel them?

While 2nd order methods have many advantages, e.g. natural gradient (e.g. in L-BFGS) attracts to close zero gradient point, which is usually saddle. Other try to pretend that our very non-convex ...
1
vote
0answers
19 views

What are good packages for online linear regression besides Vowpal Wabbit?

Does anyone know of online learning packages that implement NG and NAG algorithms from Stephen Ross' paper: chrome-extension://oemmndcbldboiebfnladdacbdfmadadm/http://auai.org/uai2013/prints/papers/...
0
votes
0answers
63 views

Multiclass hinge loss gradient

I am trying to compute the gradient of multi class hinge loss function but i am kinda confused. First things first, I have a W matrix [10xD] (10 classes) that contains the weights. The loss ...
0
votes
0answers
11 views

How to get precise answer from stochastic gradient descent

I have a convex optimization problem in few variables and I have an unbiased estimator of the gradient without having the ability to evaluate the function itself. I want to do gradient descent but the ...
4
votes
1answer
312 views

Is stochastic gradient descent pseudo-stochastic?

I know that stochastic gradient descent randomly chooses 1 sample to update the weights. An epoch is defined as using all $N$ samples. So with SGD, for each epoch, we update the weights $N$ times. ...
1
vote
0answers
46 views

How positive definite Hessian approximations for SGD (e.g. Gauss-Newton) handle saddles?

For example due to symmetry of parameters, functions optimized in machine learning usually have huge number of local minima and saddles - growing exponentially with dimension. I am trying to ...
6
votes
1answer
68 views

Do there exist adaptive step size methods for Newton-Raphson optimization?

Stochastic/Mini-batch gradient descent, caused by interest in deep learning, has made lots of advances in adaptive step sizes. For example, Adam, Nadam, Adamax, ..., are all improvements to the ...
5
votes
1answer
454 views

Why second order SGD convergence methods are unpopular for deep learning?

It seems that, especially for deep learning, there are dominating very simple methods for optimizing SGD convergence like ADAM - nice overview: http://ruder.io/optimizing-gradient-descent/ They trace ...
0
votes
1answer
20 views

Is training loss guaranteed to decrease for stochastic gradient descent?

When performing stochastic gradient descent, it is necessary for the training loss to decrease a) between iterations in an epoch? (I think the answer is no) b) between epochs? (I think the answer is ...
5
votes
0answers
17 views

Why divide the learning rate by the size of the mini batch? [duplicate]

In Michael Nielsen's online book Neural Networks and Deep Learning, in chapter one (and onwards) he divides the learning rate, $\eta$, by the size of the mini batch when he performs stochastic ...
1
vote
0answers
21 views

Tensorflow InvalidArgumentError: The determinant is not finite [closed]

I'm trying to fit a Mixture of Gaussians to a data set. First the data is clustered using K-Means Clustering. Each cluster is then fitted with a Gaussian.To avoid inversion of large covariance matrix, ...
0
votes
0answers
20 views

Accelerated Projected SGD under box constraints

Are there generalizations of ADAM or Adagrad algorithm that allow box constraints for the parameters to be incorporated in the gradient descent step? Is it valid to simply run the algorithm as usual ...
0
votes
1answer
141 views

When does my unsupervised autoencoder start to overfit?

I am working on anomaly detection using an autoencoder neural network with $1$ hidden layer. This is an unsupervised setting, as I do not have previous examples of anomalies. The input data has ...
0
votes
0answers
80 views

Natural Gradients in Stochastic Variational Inference (SVI) for Gaussian Process Regression

Currently, I've hard times in understanding the natural gradients update in SVI method for Gaussian Process. I'm learning the SVI method for Gaussian Process through Gaussian Process for Big Data ...
3
votes
1answer
238 views

Why increasing the batch size has the same effect as decaying the learning rate?

There have been a few papers this year, concerned with very large scale training, where instead than decaying the learning rate $\eta$, the batch size $B$ was increased, usually with the same schedule ...
1
vote
1answer
45 views

How to understand whether Stochastic Gradient Descent has converged?

I am using SGD to solve for MSE function. My training set is around 50K, and I am monitoring the gradient at every epoch (once a pass is completed over all the training data). I played around a lot ...
0
votes
0answers
83 views

How to choose the learning rate for stochastic gradient descent (via backtracking)?

I am trying to implement "from scratch" SGD and Mini Batch Gradient Descent in Matlab. I have to minimize a function like $f(x)= \sum f_i (X_i, y_i)$ where $(X_i, y_i)$ is a data point (features and ...
0
votes
2answers
63 views

Is downsampling okay for logistic regression if I only care about relative ordering (ROC AUC)?

I see a few discussions that suggest downsampling is never correct for logistic regression or suggesting that you have to do bias term corrections post-hoc: Downsampling vs upsampling on the ...
0
votes
0answers
104 views

need help understanding the benefit of score function estimator

The score function estimator a.k.a REINFORCE policy gradient in reinforcement learning is (from http://blog.shakirm.com/2015/11/machine-learning-trick-of-the-day-5-log-derivative-trick/): \begin{...
1
vote
0answers
790 views

Stochastic gradient descent vs mini-batch gradient descent

Gradient descent in neural networks involves the whole dataset for each weights-update step, and it is well known it would be computationally too long and also could make it converge to a local non-...
0
votes
1answer
77 views

SGD and quantile regression

It is my understanding that the quantile loss is not differentiable (at 0) so base gradient descent cannot be used. However, Vowpal Wabbit which is an SGD-based learner very much includes quantile ...
1
vote
0answers
52 views

Is it possible to combine SPSA and Adam?

In SGD algorithms such as Adam you generally make a bad estimate of the gradient of the loss function and take that gradient to move the parameters in the desired direction. Gradient free methods ...
1
vote
0answers
28 views

Stochastic gradient descent (SGD) on data with weights

Mostly deep learning model training is on data with a unit weight. In this case, every mini-batch of a fixed size, say, 32, contains exactly the same total weight (32) for each update. This is the ...
1
vote
0answers
61 views

Difference between Stochastic Gradient Descent and Sklearn's Stochastic Average Gradient (SAG) solver?

How does stochastic gradient descent varies from Sklearn's SAG (Stochastic average gradient) solver? Edit: Many sklearn models like Ridge, LogisticRegression, etc accept SAG as a solver
0
votes
1answer
456 views

stochastic gradient descent of ridge regression when regularization parameter is very big

As we know, the gradient of ridge regression is: $$ g = \frac{\partial L}{\partial \theta} = -X_i^T(y_i-X_i\theta)+2\lambda\theta $$ where $X_i$ is the $i$th training sample. The update of $\theta$ is ...
22
votes
6answers
3k views

For convex problems, does gradient in Stochastic Gradient Descent (SGD) always point at the global extreme value?

Given a convex cost function, using SGD for optimization, we will have a gradient (vector) at a certain point during the optimization process. My question is, given the point on the convex, does the ...
1
vote
1answer
183 views

Why doesn't feature standardization make SGD with momentum redundant?

In the paper An overview of gradient descent optimization algorithms, the author discusses the Momentum algorithm: SGD has trouble navigating ravines, i.e. areas where the surface curves much ...
-1
votes
1answer
60 views

Stochastic gradient descent and asymptotic analysis

In 8th chapter of deep learning book, the following lines are written under Stochastic gradient descent heading: The asymptotic analysis obscures many advantages that stochastic gradient descent ...
4
votes
1answer
534 views

Is stochastic gradient descent biased?

In the paper Mutual Information Neural Estimation, the authors derive the following gradient for the network $$ \nabla_\theta\mathcal V(\theta)=\mathbb E\left[\nabla_\theta T_\theta\right]-{\mathbb E\...
0
votes
1answer
55 views

How to deal with numeric instability in stochastic gradient descent?

Imagine that we try to perform sgd using a gradient that takes very small or very large values (e.g. it is a product of many terms that are larger than 1). Is there a standard approach to deal with ...
0
votes
2answers
75 views

Is a loss function computed after each step of gradient descent or after a whole epoch?

In neural networks with mini-batch or stochastic gradient descent, is a loss function computed after each step of gradient descent or after a whole epoch?
1
vote
0answers
84 views

Support Vector Machines: a beginner's question about the underlying math

I'm new to Support Vector Machines and I've been trying to get into the underlying math (instead of just using Scikit Learn or something like that). I understand the math behind it up to the point ...
3
votes
1answer
70 views

Regularization and weight updates in SGD

I am using single observation to compute losses using neural network implementation in PyTorch. I am confused in a small detail of SGD. If I compute loss and do ...
1
vote
0answers
158 views

Simulated Annealing vs SGD with (warm) Restarts

What's the difference between simulated annealing and stochastic gradient descent with restarts? They both seem like they are occasionally going backwards at a decreasing rate. Also what is the ...
0
votes
0answers
69 views

Is the regularization term necessary when classifying one feature?

I'm using the Stochastic Gradient Descent linear classifier (implemented in Scikit-learn) to classify an image pixel by pixel. So my dataset has only one feature, ...
0
votes
1answer
45 views

Stochastic gradient descent update

Equation 93 of Chapter 3 of Michael Nielsen's neural networks book describes the stochastic gradient descent update rule as the following: $w \leftarrow (1-\frac{\eta\lambda}{n})w - \frac{\eta}{m}\...
3
votes
1answer
355 views

Why is gradient descent with momentum considered an exponentially weighted average?

I recently watched Andrew Ng's video on SGDM. I understand that the momentum term updates the gradient by weighting the last gradient and using a small component of V_dw. I don't understand why ...
0
votes
1answer
44 views

How to define nearest neighbor search such that it can be optimized using stochastic gradient descent?

Assume that there is a reference two-dimensional array ref and a given vector x. I would like to return the closest vector to <...
7
votes
1answer
278 views

What is the difference between VAE and Stochastic Backpropagation for Deep Generative Models?

What is the difference between Auto-encoding Variational Bayes and Stochastic Backpropagation for Deep Generative Models? Does inference in both methods lead to the same results? I'm not aware of any ...
1
vote
0answers
56 views

Distribution of coefficients arrived at by stochastic gradient descent?

In SGD we have update $$w^{(k)} = w^{(k-1)} + \nabla _wL(y,w)$$ Hence $w^{(k)}$ is a sum of random variables. They're not iid, so the central limit theorem doesn't apply. Is there some result, ...