Questions tagged [stochastic-gradient-descent]
The stochastic-gradient-descent tag has no usage guidance.
206
questions
0
votes
0
answers
26
views
Convergence in Logistic Regression
Hey I'm taking a deeper dive into logistic regression. Specifically the following loss function with L2 regularization,
$$l(w)=\frac{1}{n}\sum_n \log(1+\exp(-y_i \cdot x_i^Tw))+\frac{\lambda}{2}||w||^...
- 1
1
vote
1
answer
9
views
Trying to reproduce proof of Bandit Gradient Algorithm as SGD
I'm trying to make sense of the "The Bandit Gradient Algorithm as Stochastic Gradient Ascent" proof in Sutton and Barto's intro to RL textbook. I'm stuck on the line
$E[(q_*(A_t)-B_t)\frac{\...
- 11
0
votes
0
answers
7
views
Getting SGDRegressor to converge to equivalent RidgeCV R2 results
I have a model of some financial data that achieves an R2 of ~0.01 (1%) using RidgeCV -- this is about what I expect. I'm exploring building the equivalent model using SGDRegressor so I can leverage ...
- 11
4
votes
2
answers
391
views
What is the meaning of "SGD scales the gradient uniformly in all directions"?
I'm really newbie about neural network and optimization. When I read the references, I found this journal Wang et al 2018.
The journal stated:
One disadvantage of SGD
is that it scales the gradient ...
- 117
2
votes
1
answer
57
views
What happened to the "other" stochastic gradient descent?
Stochastic gradient descent is a useful approach to improving iteration time by giving up some rate of convergence. For a parameter $w$, learning rate $\eta$, and smooth objective function $Q$ the ...
- 6,988
0
votes
0
answers
10
views
Neural Networks- What influences the choice of Optimizer more? - Specific Model Architecture or the Input Data Distribution(type, preprocessing etc)?
Basically I am implementing a CNN on custom data. I cannot go into specifics but I notice that no matter what type of model I use (For example: A custom model, or even Transfer Learning models such as ...
1
vote
0
answers
20
views
Could someone help me interpret the data that I have gathered thus far?
I am trying to train a SVM model for my statistical learning course. The problem is a binary image classification problem (wildfire, nowildfire). This is the rigorous amount of testing that I have ...
- 136
0
votes
0
answers
18
views
How can you detect if your Gradient Descent algorithm is in a local minimum and not the global minimum?
For both batch and stochastic gradient descent, how can you detect if it is stuck in a local minimum and not the global minimum?
Is there a sophisticated way to do this, as opposed to guess and check ...
- 704
2
votes
0
answers
22
views
SGD on finite datasets
In stochastic convex optimization, if
$F(w) = E[l(w^Tx,y)]$,
when l is a convex, L-Lipschitz loss function, it can be optimized using SGD such that
$E[F(\bar{w}_T)] = \frac{1}{T} E[F(w_t)] \leq \min ...
- 121
0
votes
0
answers
45
views
Why is the average used when combining the gradients of batch or mini-batch gradient descent?
When you do batch gradient descent in neural networks, you find the gradients for all the inputs in a batch and average them out. Why does the arithmetic mean work in finding the optimal gradient? Why ...
- 225
1
vote
1
answer
76
views
Deep Learning - why are we doing the forward pass on the whole dataset when using SGD
When reading the torch.optim documentation of PyTorch (https://pytorch.org/docs/master/optim.html), they do the forward pass on the whole dataset when using SGD but ...
1
vote
0
answers
231
views
Stochastic Gradient Descent and Unbiased Estimators
In stochastic gradient descent, we utilize the fact that the sampled gradient is an unbiased estimator of the full gradient. That is, with the loss as:
$ \nabla \tilde{L} = \nabla \operatorname{...
- 269
0
votes
0
answers
22
views
In stochastic approximation, does the future estimate depend on the future observation?
TL;DR: What's the correct update for stochastic approximation?
$$
\begin{aligned}
\theta_{k+1} &= \theta_k + a_{k+1} h(\theta_k, x_{k+1})\\
\text{or }\theta_{k+1} &= \theta_k + a_k h(\theta_k, ...
- 310
2
votes
1
answer
261
views
Is this "stochastic gradient method" different from the stochastic gradient descent algorithm?
In a computational statistics book, I found an optimization method to find local minimum of a function.
Let's assume that we have a differentiable function $f: \mathbb{R}^2 \longrightarrow \mathbb{R}$....
- 135
1
vote
0
answers
17
views
Why training is smoother when no randomization of batch after each epochs is applied?
I am wondering why training error converges more smoothly when I am not applying randomization after each epochs?
After each epoch I am taking the loss obtained in my last minibatch. I see that when ...
- 11
0
votes
1
answer
14
views
Loss is growing between epochs in simple model [duplicate]
I am new to ML and trying to write a very simple model that can predict the future of a function given the past. That is to say, I define a simple cubic function (y = x^3 + x^2 + x) and apply it to a ...
1
vote
1
answer
61
views
Why do we need stochastic gradient descent when we can precompute certain results in batch gradient descent?
I have read that batch gradient descent forces this summation at every step of the update, which makes it time consuming.
But if we have the following hypothesis function: $$h(x^i) = w_0 + w_1x^i$$
...
- 13
0
votes
0
answers
146
views
VGG-16 Why Validation loss is going up and down?
I am implementing VGG-16 using this cats vs dogs dataset, and am comparing the results with SGD vs Adam and getting similar results.
Here's my parameters for using Adam
...
2
votes
0
answers
297
views
Which methods can improve model fit through gradient descent?
I am looking to solve the following question:
Which methods below can improve model fit (MSE) through gradient descent?
Use a high learning rate
Use stochastic gradient descent
Start with a high ...
- 21
1
vote
1
answer
101
views
gradient ascent vs gradient descent update rule
I'm trying to understand the differences between the update rule for stochastic gradient ascent and descent. I've read some articles and still don't understand how to calculate the update rule:
...
- 11
6
votes
2
answers
656
views
ALS vs SGD in parallelization
So given the standard objective in matrix factorization for collaborative filtering of minimizing:
$$
L = \sum_{u,i \in S} (r_{ui}-q_i^Tp_u)^2 + \lambda(\sum_i||q_i^2||+\sum_u||p_u^2||)
$$
, where $r_{...
- 321
4
votes
1
answer
342
views
Backpropagation in mini-batch stochastic gradient descent with mean squared error loss
Suppose I have an ANN which has one input layer of size $128$, one hidden layer of size $64$ and one output layer of size $10$ for a classification problem. Let's assume we have a training sample of $...
- 143
1
vote
0
answers
95
views
Which is better: performance stability or high performance in the initial epochs but extremely unstable performance during validation?
I am working on a multiclassification problem using time series data. I am using a hybrid model (such as LSTM, CNN, attention, etc.). I tried two optimizers, ADAM with 0.001 learning rate and SGD with ...
- 161
1
vote
1
answer
81
views
details of stochastic Expectation Maximization (EM) algorithm
I went into a paper, Online EM Algorithm for Latent Data Models (Olivier Cappé & Eric Moulines, 2009). I got confused by the first equation the authors wrote, the Q function:
Here the authors ...
- 11
3
votes
1
answer
31
views
What could go wrong if I do gradient descent by class?
For example, when training on CIFAR10, each minibatch typically contains images from all 10 classes (assuming a moderately large batch size such as 64). What could go wrong if I train on a homogeneous ...
- 1,727
3
votes
3
answers
617
views
How to find the gradient when a black box I/O function is involved in evaluation of the loss?
I am trying to learn a neural network $NN_\pi$ to minimize the loss function
$$ L_{\pi} = || Y_{true} - F(X_{true}, NN_{\pi}(X_{true}) ) ||^2 $$
where $F$ is a black box (I/O) function (we only ...
- 59
2
votes
1
answer
352
views
How to handle weighted examples in stochastic gradient descent (with mini-batches)?
Suppose I have $M$ data points $x_i$ and associated weights $w_i > 0$. I want to optimize a function,
$$F(\theta) = \frac{1}{M}\sum_i w_i f(x_i;\theta)$$
in the parameters $\theta$. I will assume ...
- 3,562
1
vote
0
answers
108
views
Regular *negative* spikes in neural network training loss
I am training an ALL-CNN network using the Adam solver. As the figure shows, the testing seems to converge to an acceptable solution, but there are these regular negative spikes during training that ...
- 11
2
votes
0
answers
175
views
Mini-Batch Gradient Descent - Large batch size require small learning rate?
Coursera Machine Learning in the Enterprise - Science of Machine Learning and Custom Training says large batch size require smaller LR.
However, How should the learning rate change as the batch size ...
- 1,334
1
vote
0
answers
183
views
Derive the gradient matrices w.r.t. W1 and W2 and backprop equation in a Residual Network [closed]
How would I go about deriving gradient matrices w.r.t. W1 and W2 and backpropagation equation in a residual block that is a part of a larger ResNet network with forward propagation expressed as:
$$ F(...
- 11
1
vote
1
answer
120
views
How does SGD training error decrease in subsequent epochs with non-iid samples when it is recommended that samples in subsequent epochs be iid?
I have been reading the Deep Learning book by Ian Goodfellow and on pg. 277, they mention:
It is also crucial that the minibatches be selected randomly.
Computing an unbiased estimate of the expected ...
- 131
2
votes
1
answer
320
views
which training mode is more convenient for small datasets?
I have a regression problem to be solved using one of neural networks models, but I have a small dataset which contains 30 samples.
Which training mode is more suitable for such dataset: stochastic or ...
- 153
2
votes
0
answers
153
views
Bias introduced when using weak shuffling
I have batch learning problem (in this particular case a neural network) where I am training my data in batches, and then repeating for a number of epochs.
In Stochastic Gradient Descent, we minimise ...
- 63
0
votes
1
answer
115
views
About update procedure in data incremental learning
As far as I understood, the idea of data incremental learning consists of keeping the model always up to date. Suppose that we trained a model for user recognition using voice as input. Therefore, the ...
- 171
3
votes
1
answer
840
views
Why does ADAM optimization perform well on non-convex functions and bad on convex functions?
I'm currently trying to understand SGD and ADAM optimization, and I understand that ADAM optimization performs well on non-convex loss functions and that SGD performs well on convex loss functions (...
- 237
1
vote
0
answers
228
views
Does always gradients in mini-batch SGD have to be unbiased in order to prove convergence?
I am currently reading this paper [1] and [2].
The authors state that:
Our analytical results include almost all of the unbiased compression
techniques.
And also:
(i) gradient compression must be ...
- 43
3
votes
1
answer
119
views
Stochastic Gradient Descent Code Check for Least Squares
I have an R-based implementation of the gradient descent and am trying to also get it to work as SGD. The function matches R's lm function when using the entire data set. But, when I sample from the ...
- 193
1
vote
1
answer
185
views
Performing Linear Regression using Stochastic Gradient Descent, by batches
I am presented with a data set, where I am supposed to perform linear regression on this using SGD. My first instinct would be to train each data point there is until I reach the last one. Only then ...
- 8,647
4
votes
1
answer
675
views
Why is a 2nd order derivative optimization better for no hidden layer neural networks?
I was reading in this blog. That first order derivative SGD optimization methods are worse for neural networks without hidden layers and 2nd order is better, because that's what regression uses. Why ...
- 710
0
votes
1
answer
361
views
Why does stochastic gradient descent lead us to a minimum at all?
Why do we think that stochastic gradient descent is going to find a minimum at all? I mean on each iteration SGD moves in the direction that reduces only current batch's error (SGD doesn't care about ...
- 531
3
votes
1
answer
514
views
The reason (and intuition) behind why stochastic gradient descent can get stuck on a local minimum
Suppose you want to find $k$ that minimises your cost function $J(k)$. We may want to apply batch gradient descent or stochastic gradient descent. Let's deliberately initialise $k$ with the same ...
- 531
0
votes
0
answers
24
views
Can stochastic gradient descent for Bayesian Inference? [duplicate]
I was looking at the Bayesian MAP estimate formula which is the "argmax(likelihood * prior)".
Can this be calculated using stochastic gradient descent? Gradient descent requires knowing the ...
1
vote
1
answer
124
views
Is outer product of marginal distribution the "best" mean-field approximation for a joint distribution?
I am certain this has been asked somewhere else, if that's the case, link me and close the thread.
I am studying variational inference and mean-field approximation. All the explanations I come across ...
- 113
2
votes
1
answer
34
views
Why a minimiser of a subset of training dataset is that of the whole training set
In section 3.3 of Bottou et al (2018), under the 'intuitive motivation' paragraph, the authors claim that 'a minimiser of empirical risk for the larger set $S$ is clearly given by a minimiser for the ...
- 330
3
votes
1
answer
635
views
upsampling vs class weights in mini-batch SGD
Let's consider using mini-batch SGD in (neural network) binary classification problem with imbalanced dataset. Let's say that the ratio between the number of examples in each class is positive:...
- 51
1
vote
0
answers
2k
views
When are very small learning rates useful?
I just wondered if there are cases where small or very small learning rates in gradient descent based optimization are useful?
A large learning rate allows the model to explore a much larger portion ...
- 645
0
votes
0
answers
54
views
Gradient Descent Algorithm for Interdependent parameters
Suppose I have $n$ data points ($X_i$,$y_i$) where $X_i$ is a vector and $y_i$ is a scalar, $1 \le i \le n$. By defining $\hat{\boldsymbol{Y}} = \boldsymbol{\Theta} \boldsymbol{X} + \boldsymbol{b}$ ...
- 65
0
votes
1
answer
5k
views
Loss values above 1.0
I have a convolutional neural network for tensors classification in Pytorch. I am using Cross-Entropy Loss. My optimizer is Stochastic Gradient Descent and the learning rate is 0.0001. The accuracy of ...
- 41
3
votes
0
answers
180
views
Why adversarial training prefer SGD over Adam?
Why adversarial training methods (e.g. Trades) use SGD as optimizer rather than Adam?
- 31
3
votes
1
answer
673
views
Relationship between variance of gradient and SGD convergence
I've found things such as the Robbins-Monroe conditions for the learning rate, as well as a proof from Robbins, Siegmund, 1971 which gives convergence to a local minima provided that the expectation ...
- 208