The tag has no usage guidance.

60 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
724 views

As is known to all, stochastic gradient descent is a popular optimizer in machine learning. There have been many variants of SGD. However, it has come to my attention that no one talks about the ...
68 views

### Beneficial dimension for 2nd order modelling in SGD optimization?

There are currently mostly used first order methods in SGD optimizers, second order are often seen too costly as e.g. full Hessian has size $D^2$ in dimension $D$. But we don't need full Hessian - ...
335 views

### Difference between Stochastic Approximation (SA) and Stochastic Gradient Descent (SGD)

I understand the intended use cases for both stochastic approximation algorithms like SPSA or FDSA, and for SGD algorithms like Adam. SPSA is intended for noisy objective functions, and Adam for ...
515 views

### Counter intuitive behavior from scikit-learn's SGDClassifier

I am working with SGDClassifier from Python library scikit-learn, a function which implements linear classification with a ...
45 views

### How does scaled conjugate gradient work in neural network training? Comparison with gradient descent

I am very new and beginner in the machine learning world, and I would like to ask if someone could simply explain to me how does the scaled conjugate gradient method work in neural network training? ...
2k views

Gradient descent in neural networks involves the whole dataset for each weights-update step, and it is well known it would be computationally too long and also could make it converge to a local non-...
247 views

### Support Vector Machines: a beginner's question about the underlying math

I'm new to Support Vector Machines and I've been trying to get into the underlying math (instead of just using Scikit Learn or something like that). I understand the math behind it up to the point ...
154 views

### Is mini-batch / stochastic gradient descend similar implicitly adding the same effect as simulated annealing?

Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak Peter Tang. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. https://arxiv.org/...
282 views

### Convergence of gradient descent Monte Carlo Control with function approximation

Can anyone point me in the direction of a formal proof of convergence for a (on/off policy) Monte Carlo control algorithm with (non-)linear function approximation? In http://incompleteideas.net/book/...
140 views

130 views

### Is it possible to combine SPSA and Adam?

In SGD algorithms such as Adam you generally make a bad estimate of the gradient of the loss function and take that gradient to move the parameters in the desired direction. Gradient free methods ...
96 views

### Stochastic gradient descent (SGD) on data with weights

Mostly deep learning model training is on data with a unit weight. In this case, every mini-batch of a fixed size, say, 32, contains exactly the same total weight (32) for each update. This is the ...
142 views

### Difference between Stochastic Gradient Descent and Sklearn's Stochastic Average Gradient (SAG) solver?

How does stochastic gradient descent varies from Sklearn's SAG (Stochastic average gradient) solver? Edit: Many sklearn models like Ridge, LogisticRegression, etc accept SAG as a solver
344 views

### Simulated Annealing vs SGD with (warm) Restarts

What's the difference between simulated annealing and stochastic gradient descent with restarts? They both seem like they are occasionally going backwards at a decreasing rate. Also what is the ...
114 views

### SGD shows the same convergence behaviour as batch gradient descent when using adaptive learning rate?

SGD shows the same convergence behaviour as batch gradient descent when using adaptive learning rate ? I dont understand why he claimed that. I couldnt find any reference about it in any paper. ...
1k views

### Recommender Systems with Implicit Feedback Data - Modelling and Updating

I have a very large dataset of user-item interactions. It tells me how many times a user bought, viewed or liked an item - all these actions are in binary form and I don't have any explicit ratings. ...
1k views

### Ridge regression using stochastic gradient descent in Python

I am trying to implement a solution to Ridge regression in Python using Stochastic gradient descent as the solver. My code for SGD is as follows: ...
378 views

### How to get SGD to reach global optimal point in logistic regression?

I am trying to write a tool which involves implementing logistic regression. With the batch gradient descent method, the convergence is guaranteed as it is a convex problem. However, I find that with ...
48 views

### SGD unbiased estimator: 1 example vs larger minibatch for each iteration

Studying the SGD, I found that at each iteration the SGD turns out to be an unbiased estimator of the full gradients. The number of iterations (stochastic gradient estimation) depends on the variance. ...
139 views

I'm reading Hands-On Machine Learning with Scikit-Learn, Keras & Tensorflow and on page 325 (follows up on 326) there's a following piece of text on learning-rate: The learning is arguably the ...
41 views

### How can I improve a classification algorithm for dogs and cats?

The following code is a ML algorithm trained to classify between dogs and cats, the database is composed by 25000 images (evenly split) and can be obtained at this Link (if you click it will ...
37 views

### Batch size influence on model quality

I found in https://dl.acm.org/doi/abs/10.1145/3320060 (section 3) this graph that illustrates influence of batch size. Below there is an explanation: We can show the existence of region C by ...
86 views

### On projected gradient descent and inequality constraints

Consider the optimization problem $$\min_{x\in\mathbb{R}^n} \quad f(x)$$ using the gradient descent, we can iteratively solve this problem x^{k+1} = x^k-\...
21 views

### Adam converges while SGD does not improve at all

I am trying to build a model based movie recommendation system with a neural network. The architecture looks as follows: ...
75 views

### Neural network doesn't converge but has good performance

I have a sequence (> 100 million) of symbols and several models predict the next symbol. To combine these predictions I'm using stacked generalization with a multilayer perceptron trained with online ...
228 views

### CNN training loss regular spikes at the end of the epoch

I am training a CNN in PyTorch with Adam and the initial learning rate is 1e-5. I have 5039 samples in my epoch and the batch size is 1. I have observed that I have a regular spike pattern of training ...
51 views

### Comparing numerical stability and computing bounds on the condition number of learned weights

I have an empirical risk minimization problem with two equivalent losses that solves it, $f_1(x; \theta_1)$ and $f_2(x ; \theta_2)$, where $x$ is the data and $\theta$ are the model parameters (in ...
74 views

640 views

### Hinton claims SGD with batch norm can help: How?

In Hinton's paper "Layer Normalization", on the first page he says Feedforward neural networks trained using batch normalization converge faster even with simple SGD. By this I think he means ...
177 views

15 views

### SGD with momentum update - CS231n explanation

In the notes for Stanford's CS231n course there is an explanation for the Momentum update. I'm confused by the usage of the word "integrates" here, e.g. "gradient directly integrates ...
5 views

### Epoch Metrics vs. Step Metrics?

I'm running PyTorch and I'm trying to log metrics. I just have one question - When using mini-batch gradient descent, and logging metrics in the training loop, you can get the training and validation ...