Questions tagged [stochastic-gradient-descent]

60 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5
votes
0answers
724 views

Why no one talks about stochastic conjugate gradient descent?

As is known to all, stochastic gradient descent is a popular optimizer in machine learning. There have been many variants of SGD. However, it has come to my attention that no one talks about the ...
4
votes
0answers
68 views

Beneficial dimension for 2nd order modelling in SGD optimization?

There are currently mostly used first order methods in SGD optimizers, second order are often seen too costly as e.g. full Hessian has size $D^2$ in dimension $D$. But we don't need full Hessian - ...
4
votes
0answers
335 views

Difference between Stochastic Approximation (SA) and Stochastic Gradient Descent (SGD)

I understand the intended use cases for both stochastic approximation algorithms like SPSA or FDSA, and for SGD algorithms like Adam. SPSA is intended for noisy objective functions, and Adam for ...
4
votes
0answers
515 views

Counter intuitive behavior from scikit-learn's SGDClassifier

I am working with SGDClassifier from Python library scikit-learn, a function which implements linear classification with a ...
3
votes
0answers
45 views

How does scaled conjugate gradient work in neural network training? Comparison with gradient descent

I am very new and beginner in the machine learning world, and I would like to ask if someone could simply explain to me how does the scaled conjugate gradient method work in neural network training? ...
3
votes
0answers
2k views

Stochastic gradient descent vs mini-batch gradient descent

Gradient descent in neural networks involves the whole dataset for each weights-update step, and it is well known it would be computationally too long and also could make it converge to a local non-...
3
votes
0answers
247 views

Support Vector Machines: a beginner's question about the underlying math

I'm new to Support Vector Machines and I've been trying to get into the underlying math (instead of just using Scikit Learn or something like that). I understand the math behind it up to the point ...
3
votes
0answers
154 views

Is mini-batch / stochastic gradient descend similar implicitly adding the same effect as simulated annealing?

Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak Peter Tang. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. https://arxiv.org/...
3
votes
0answers
282 views

Convergence of gradient descent Monte Carlo Control with function approximation

Can anyone point me in the direction of a formal proof of convergence for a (on/off policy) Monte Carlo control algorithm with (non-)linear function approximation? In http://incompleteideas.net/book/...
3
votes
1answer
140 views

constant terms in stochastic gradient descent: when to apply them and how much of the constant gradient component?

in a derivation for the gradient of a collaborative filtering system (similar to Probabilistic Matrix Factorization), I got to the following expression for the gradient of a latent vector $\mathbf{u}...
2
votes
1answer
107 views

Can you use stochastic gradient descent with a multinomial likelihood?

I have a multinomial likelihood of the form: $$P(\underline n|\underline x) = N!\prod_{i=1}^M \frac{f_i(\underline x)^{n_i}}{n_i!}$$ where $\underline x$ is a vector of parameters, $f_i(\underline x)...
2
votes
0answers
130 views

Is it possible to combine SPSA and Adam?

In SGD algorithms such as Adam you generally make a bad estimate of the gradient of the loss function and take that gradient to move the parameters in the desired direction. Gradient free methods ...
2
votes
0answers
96 views

Stochastic gradient descent (SGD) on data with weights

Mostly deep learning model training is on data with a unit weight. In this case, every mini-batch of a fixed size, say, 32, contains exactly the same total weight (32) for each update. This is the ...
2
votes
0answers
142 views

Difference between Stochastic Gradient Descent and Sklearn's Stochastic Average Gradient (SAG) solver?

How does stochastic gradient descent varies from Sklearn's SAG (Stochastic average gradient) solver? Edit: Many sklearn models like Ridge, LogisticRegression, etc accept SAG as a solver
2
votes
0answers
344 views

Simulated Annealing vs SGD with (warm) Restarts

What's the difference between simulated annealing and stochastic gradient descent with restarts? They both seem like they are occasionally going backwards at a decreasing rate. Also what is the ...
2
votes
0answers
114 views

SGD shows the same convergence behaviour as batch gradient descent when using adaptive learning rate?

SGD shows the same convergence behaviour as batch gradient descent when using adaptive learning rate ? I dont understand why he claimed that. I couldnt find any reference about it in any paper. ...
2
votes
0answers
1k views

Recommender Systems with Implicit Feedback Data - Modelling and Updating

I have a very large dataset of user-item interactions. It tells me how many times a user bought, viewed or liked an item - all these actions are in binary form and I don't have any explicit ratings. ...
2
votes
0answers
1k views

Ridge regression using stochastic gradient descent in Python

I am trying to implement a solution to Ridge regression in Python using Stochastic gradient descent as the solver. My code for SGD is as follows: ...
2
votes
0answers
378 views

How to get SGD to reach global optimal point in logistic regression?

I am trying to write a tool which involves implementing logistic regression. With the batch gradient descent method, the convergence is guaranteed as it is a convex problem. However, I find that with ...
1
vote
1answer
48 views

SGD unbiased estimator: 1 example vs larger minibatch for each iteration

Studying the SGD, I found that at each iteration the SGD turns out to be an unbiased estimator of the full gradients. The number of iterations (stochastic gradient estimation) depends on the variance. ...
1
vote
1answer
139 views

Adam (adaptive) optimizer(s) learning rate tuning

I'm reading Hands-On Machine Learning with Scikit-Learn, Keras & Tensorflow and on page 325 (follows up on 326) there's a following piece of text on learning-rate: The learning is arguably the ...
1
vote
1answer
41 views

How can I improve a classification algorithm for dogs and cats?

The following code is a ML algorithm trained to classify between dogs and cats, the database is composed by 25000 images (evenly split) and can be obtained at this Link (if you click it will ...
1
vote
0answers
37 views

Batch size influence on model quality

I found in https://dl.acm.org/doi/abs/10.1145/3320060 (section 3) this graph that illustrates influence of batch size. Below there is an explanation: We can show the existence of region C by ...
1
vote
0answers
86 views

On projected gradient descent and inequality constraints

Consider the optimization problem \begin{equation} \min_{x\in\mathbb{R}^n} \quad f(x) \end{equation} using the gradient descent, we can iteratively solve this problem \begin{equation} x^{k+1} = x^k-\...
1
vote
0answers
21 views

Adam converges while SGD does not improve at all

I am trying to build a model based movie recommendation system with a neural network. The architecture looks as follows: ...
1
vote
0answers
75 views

Neural network doesn't converge but has good performance

I have a sequence (> 100 million) of symbols and several models predict the next symbol. To combine these predictions I'm using stacked generalization with a multilayer perceptron trained with online ...
1
vote
0answers
228 views

CNN training loss regular spikes at the end of the epoch

I am training a CNN in PyTorch with Adam and the initial learning rate is 1e-5. I have 5039 samples in my epoch and the batch size is 1. I have observed that I have a regular spike pattern of training ...
1
vote
0answers
51 views

Comparing numerical stability and computing bounds on the condition number of learned weights

I have an empirical risk minimization problem with two equivalent losses that solves it, $f_1(x; \theta_1)$ and $f_2(x ; \theta_2)$, where $x$ is the data and $\theta$ are the model parameters (in ...
1
vote
0answers
74 views

Choosing learning rate with 2nd order method - minimizing parabola in one step?

In parabola $(\theta,g)$ values are in line $(g=f'(\theta))$ - we can get slope of this line e.g. by dividing their standard deviations: $$ \mu = \frac{\sigma_\theta}{\sigma_g}=\sqrt{\frac{var(\...
1
vote
0answers
17 views

Modified Loss Function(s) for decorrelating neurons within a layer?

I'm looking for previous references on a specific topic. Does anyone know of any modified loss functions that incentivize a network to produce a diagonal neuron-to-neuron covariance matrix (averaging ...
1
vote
0answers
795 views

Time complexity of batch gradient descent

I am read http://papers.nips.cc/paper/4937-accelerating-stochastic-gradient-descent-using-predictive-variance-reduction.pdf paper. It states that "Due to the poor condition number, the standard batch ...
1
vote
0answers
17 views

Why picking several times of same instances generally converge faster than going through instance by instance using Stochastic Gradient Descent?

I am reading Hands-on Machine Learning with Scikit-Learn & TensorFlow by Aurelien Geron. In chapter 4: Training models page 122, where it is explaining linear regression using SGD, it says that ...
1
vote
0answers
76 views

Momentum updates average of $g$, Adagrad also of $g^2$ - any other interesting updated averages for SGD convergence?

Updating exponential moving average is a basic tool of SGD methods, starting with of gradient $g$ in momentum method to extract local linear trend from the statistics. Then e.g. Adagrad, ADAM family ...
1
vote
0answers
46 views

What are good packages for online linear regression besides Vowpal Wabbit?

Does anyone know of online learning packages that implement NG and NAG algorithms from Stephen Ross' paper: chrome-extension://oemmndcbldboiebfnladdacbdfmadadm/http://auai.org/uai2013/prints/papers/...
1
vote
0answers
190 views

How positive definite Hessian approximations for SGD (e.g. Gauss-Newton) handle saddles?

For example due to symmetry of parameters, functions optimized in machine learning usually have huge number of local minima and saddles - growing exponentially with dimension. I am trying to ...
1
vote
0answers
122 views

Distribution of coefficients arrived at by stochastic gradient descent?

In SGD we have update $$w^{(k)} = w^{(k-1)} + \nabla _wL(y,w)$$ Hence $w^{(k)}$ is a sum of random variables. They're not iid, so the central limit theorem doesn't apply. Is there some result, ...
1
vote
0answers
640 views

Hinton claims SGD with batch norm can help: How?

In Hinton's paper "Layer Normalization", on the first page he says Feedforward neural networks trained using batch normalization converge faster even with simple SGD. By this I think he means ...
1
vote
0answers
177 views

Behavior of AdaGrad without the square root in the denominator

Multiple articles claim that AdaGrad does not work well when the square-root in the formula is not taken. This is one such example. $\theta_{t+1,i} = \theta_{t,i}-\dfrac{\eta}{\sqrt{G_{t,ii}+\epsilon}...
1
vote
0answers
201 views

Why does Adagrad improve the robustness of SGD?

I mainly read this blog. And this blog sites this paper for the statement that Adagrad improved the robustness of SGD. I have tried to check the original paper or other articles which explains why ...
1
vote
0answers
585 views

Loss update with mini batches and after epoch

If I understand correctly, when using deep learning with mini batches, we have a forward and backward pass in every mini batch (with the corresponding optimizer). But does something different happen ...
1
vote
0answers
112 views

Regression for Feature Importance

I think the answer to this question should be simple but I have searched the internet for a while and not found anything. I have a set of categorical attributes and a continuous target variable. My ...
1
vote
0answers
138 views

What cause $X\beta$ shift from Stochastic Gradient Descent Comparing to Logistic Regression?

I am experimenting with stochastic gradient descent and observing very strange output. In a toy problem, the $X\beta$ for stochastic gradient descent is always larger than $0$, which will be ...
1
vote
0answers
413 views

The Convergence Plot in Stochastic Gradient Descent (SGD)

Q1: The main question is that can I say my own coded SGD algorithm converged based on the convergence plot below? In the SGD code, the data is randomly shuffled in each epoch before calculating the ...
1
vote
0answers
393 views

Convergence of validation error on MNIST dataset with RMSProp and Adam

(This is my first question so apologies if I get something wrong/am not clear enough) As part of a school assignment, I'm testing the rate of convergence and final results of training a neural ...
1
vote
1answer
244 views

With SGD, how to decide the number of steps to train?

I'm taking the Udacity/Google's Deep Learning course. For problem set 2, we are training an SGD model. One can tune the hyper-parameters (batch_size, number of ...
0
votes
0answers
25 views

Least squares fit with polynomials: Order of the polynomial vs accuracy of the predictions

I have noisy evaluations $y_i$ of some unkown function $f$ at points $\vec{x}_i$ clustered around point $\vec{x^*}$. Now I want to fit a polynomial model to this data to get some surrogate model $\hat{...
0
votes
0answers
15 views

SGD with momentum update - CS231n explanation

In the notes for Stanford's CS231n course there is an explanation for the Momentum update. I'm confused by the usage of the word "integrates" here, e.g. "gradient directly integrates ...
0
votes
0answers
5 views

Epoch Metrics vs. Step Metrics?

I'm running PyTorch and I'm trying to log metrics. I just have one question - When using mini-batch gradient descent, and logging metrics in the training loop, you can get the training and validation ...
0
votes
0answers
33 views

Implementing Stochastic Gradient Descent with both Weight Decay and Momentum

So I'm trying to implement a neural network using only numpy module in Python. The problem I'm facing is related to the correct implementation of the regularization through weight decay, and also the ...
0
votes
0answers
19 views

When is a high learning rate for Stochastic Gradient Descent a good thing?

I was always under the impression that SGD needed a lower learning rate than optimizers like Adam, because it was stochastic and more likely to make training diverge with higher learning rates. I ...