# Questions tagged [stochastic-gradient-descent]

The tag has no usage guidance.

60 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
0answers
724 views

### Why no one talks about stochastic conjugate gradient descent?

As is known to all, stochastic gradient descent is a popular optimizer in machine learning. There have been many variants of SGD. However, it has come to my attention that no one talks about the ...
0answers
68 views

### Beneficial dimension for 2nd order modelling in SGD optimization?

There are currently mostly used first order methods in SGD optimizers, second order are often seen too costly as e.g. full Hessian has size $D^2$ in dimension $D$. But we don't need full Hessian - ...
0answers
335 views

### Difference between Stochastic Approximation (SA) and Stochastic Gradient Descent (SGD)

I understand the intended use cases for both stochastic approximation algorithms like SPSA or FDSA, and for SGD algorithms like Adam. SPSA is intended for noisy objective functions, and Adam for ...
0answers
515 views

### Counter intuitive behavior from scikit-learn's SGDClassifier

I am working with SGDClassifier from Python library scikit-learn, a function which implements linear classification with a ...
0answers
45 views

### How does scaled conjugate gradient work in neural network training? Comparison with gradient descent

I am very new and beginner in the machine learning world, and I would like to ask if someone could simply explain to me how does the scaled conjugate gradient method work in neural network training? ...
0answers
2k views

### Stochastic gradient descent vs mini-batch gradient descent

Gradient descent in neural networks involves the whole dataset for each weights-update step, and it is well known it would be computationally too long and also could make it converge to a local non-...
0answers
247 views

### Support Vector Machines: a beginner's question about the underlying math

I'm new to Support Vector Machines and I've been trying to get into the underlying math (instead of just using Scikit Learn or something like that). I understand the math behind it up to the point ...
0answers
154 views

### Is mini-batch / stochastic gradient descend similar implicitly adding the same effect as simulated annealing?

Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak Peter Tang. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. https://arxiv.org/...
0answers
282 views

### Convergence of gradient descent Monte Carlo Control with function approximation

Can anyone point me in the direction of a formal proof of convergence for a (on/off policy) Monte Carlo control algorithm with (non-)linear function approximation? In http://incompleteideas.net/book/...
1answer
140 views

0answers
130 views

### Is it possible to combine SPSA and Adam?

In SGD algorithms such as Adam you generally make a bad estimate of the gradient of the loss function and take that gradient to move the parameters in the desired direction. Gradient free methods ...
0answers
96 views

### Stochastic gradient descent (SGD) on data with weights

Mostly deep learning model training is on data with a unit weight. In this case, every mini-batch of a fixed size, say, 32, contains exactly the same total weight (32) for each update. This is the ...
0answers
142 views

### Difference between Stochastic Gradient Descent and Sklearn's Stochastic Average Gradient (SAG) solver?

How does stochastic gradient descent varies from Sklearn's SAG (Stochastic average gradient) solver? Edit: Many sklearn models like Ridge, LogisticRegression, etc accept SAG as a solver
0answers
344 views

### Simulated Annealing vs SGD with (warm) Restarts

What's the difference between simulated annealing and stochastic gradient descent with restarts? They both seem like they are occasionally going backwards at a decreasing rate. Also what is the ...
0answers
114 views

### SGD shows the same convergence behaviour as batch gradient descent when using adaptive learning rate?

SGD shows the same convergence behaviour as batch gradient descent when using adaptive learning rate ? I dont understand why he claimed that. I couldnt find any reference about it in any paper. ...
0answers
1k views

### Recommender Systems with Implicit Feedback Data - Modelling and Updating

I have a very large dataset of user-item interactions. It tells me how many times a user bought, viewed or liked an item - all these actions are in binary form and I don't have any explicit ratings. ...
0answers
1k views

### Ridge regression using stochastic gradient descent in Python

I am trying to implement a solution to Ridge regression in Python using Stochastic gradient descent as the solver. My code for SGD is as follows: ...
0answers
378 views

### How to get SGD to reach global optimal point in logistic regression?

I am trying to write a tool which involves implementing logistic regression. With the batch gradient descent method, the convergence is guaranteed as it is a convex problem. However, I find that with ...
1answer
48 views

### SGD unbiased estimator: 1 example vs larger minibatch for each iteration

Studying the SGD, I found that at each iteration the SGD turns out to be an unbiased estimator of the full gradients. The number of iterations (stochastic gradient estimation) depends on the variance. ...
1answer
139 views

### Adam (adaptive) optimizer(s) learning rate tuning

I'm reading Hands-On Machine Learning with Scikit-Learn, Keras & Tensorflow and on page 325 (follows up on 326) there's a following piece of text on learning-rate: The learning is arguably the ...
1answer
41 views

### How can I improve a classification algorithm for dogs and cats?

The following code is a ML algorithm trained to classify between dogs and cats, the database is composed by 25000 images (evenly split) and can be obtained at this Link (if you click it will ...
0answers
37 views

### Batch size influence on model quality

I found in https://dl.acm.org/doi/abs/10.1145/3320060 (section 3) this graph that illustrates influence of batch size. Below there is an explanation: We can show the existence of region C by ...
0answers
86 views

### On projected gradient descent and inequality constraints

Consider the optimization problem \begin{equation} \min_{x\in\mathbb{R}^n} \quad f(x) \end{equation} using the gradient descent, we can iteratively solve this problem \begin{equation} x^{k+1} = x^k-\...
0answers
21 views

### Adam converges while SGD does not improve at all

I am trying to build a model based movie recommendation system with a neural network. The architecture looks as follows: ...
0answers
75 views

### Neural network doesn't converge but has good performance

I have a sequence (> 100 million) of symbols and several models predict the next symbol. To combine these predictions I'm using stacked generalization with a multilayer perceptron trained with online ...
0answers
228 views

### CNN training loss regular spikes at the end of the epoch

I am training a CNN in PyTorch with Adam and the initial learning rate is 1e-5. I have 5039 samples in my epoch and the batch size is 1. I have observed that I have a regular spike pattern of training ...
0answers
51 views

### Comparing numerical stability and computing bounds on the condition number of learned weights

I have an empirical risk minimization problem with two equivalent losses that solves it, $f_1(x; \theta_1)$ and $f_2(x ; \theta_2)$, where $x$ is the data and $\theta$ are the model parameters (in ...
0answers
74 views

0answers
640 views

### Hinton claims SGD with batch norm can help: How?

In Hinton's paper "Layer Normalization", on the first page he says Feedforward neural networks trained using batch normalization converge faster even with simple SGD. By this I think he means ...
0answers
177 views

0answers
15 views

### SGD with momentum update - CS231n explanation

In the notes for Stanford's CS231n course there is an explanation for the Momentum update. I'm confused by the usage of the word "integrates" here, e.g. "gradient directly integrates ...
0answers
5 views

### Epoch Metrics vs. Step Metrics?

I'm running PyTorch and I'm trying to log metrics. I just have one question - When using mini-batch gradient descent, and logging metrics in the training loop, you can get the training and validation ...
0answers
33 views

### Implementing Stochastic Gradient Descent with both Weight Decay and Momentum

So I'm trying to implement a neural network using only numpy module in Python. The problem I'm facing is related to the correct implementation of the regularization through weight decay, and also the ...
0answers
19 views

### When is a high learning rate for Stochastic Gradient Descent a good thing?

I was always under the impression that SGD needed a lower learning rate than optimizers like Adam, because it was stochastic and more likely to make training diverge with higher learning rates. I ...