Skip to main content

Questions tagged [stochastic-gradient-descent]

Filter by
Sorted by
Tagged with
1 vote
0 answers
17 views

SVRG vs full gradient descent

Stochastic gradient descent allows us to avoid the computation of full gradients at the expense of introducing a noise floor to convergence. To decrease this noise floor, SGD requires a decrease in ...
hegash's user avatar
  • 111
0 votes
0 answers
11 views

EWMA formula for SGD with momentum different than generic EWMA formula

I am currently trying to understand how SGD with momentum works, what I understand is it uses the Exponential Weighted Moving Average concept to make the updates smoother. We take weighted average of ...
learnToCode's user avatar
0 votes
0 answers
11 views

Is there room for finding a more efficient hybrid optimization problem, in the context of optimization algorithms for MLE?

Recently finished my statistical modelling class, but it only briefly touched on Maximum Likelihood Estimates and I thought it was an interesting topic, so I decided to go deeper in my own time. I ...
Kevin's user avatar
  • 1
0 votes
0 answers
17 views

Error term in SGD with momentum

I am reading the article "How Momentum really works" (https://distill.pub/2017/momentum/), and i am confused in one point: I am trying to derive the convergence rate for momentum from the ...
Patricio's user avatar
  • 121
0 votes
1 answer
26 views

Question on the Partial Derivative of the Cross-Entropy Loss in SGD for Neural Networks

I'm currently learning about neural networks and stumbled upon a confusion related to the use of Stochastic Gradient Descent (SGD) in training. Specifically, I'm puzzled about the computation of the ...
user avatar
0 votes
0 answers
14 views

Neural net and category separation with interaction terms

We have a stream of tabular data consiting of categorical and numerical features. One category is somehow crucial in either affecting the target, interacting with other features and sorting the data ...
funkymind's user avatar
0 votes
0 answers
57 views

Huber-Loss optimisation using Stochastic Gradient Descent to estimate intercept and coefficient of regression line

What: I'm trying to minimise the Huber-Loss for a linear regression using Stochastic Gradient Descent from scratch. Problem: It seems like that the coeffcient $m$ doesn't get optimised, therefore the ...
Corbjn's user avatar
  • 111
0 votes
0 answers
50 views

Simple RNN for predicting the next character [duplicate]

I implemented a simple RNN from scratch (using only the numpy library )for predicting the next characters, and I trained it on a simple text=“hello world”. It works ...
user avatar
0 votes
0 answers
22 views

Understanding "Understanding the difficulty of training deep feedforward neural networks"

I'm following up on this question with a slightly more specific clarification I'd like to have addressed. I'm well familiar with covariance matrices as a matrix-valued generalization to random vectors ...
Chris's user avatar
  • 103
0 votes
0 answers
65 views

Clarifying the arguments of "Understanding the difficulty of training deep feedforward neural networks"

EDIT: Following on the comment of Sycorax, I am assuming that equation (4) is an "immediate consequence" of assuming the relative linearity of $f$ under the "regime" of our inputs. ...
Chris's user avatar
  • 103
0 votes
0 answers
10 views

Update rule with gradient of loss AND loss itself multiplied

I was reading a paper about Neural Holography (page 5, equation 4), where authors used simple stochastic gradient descent as optimizing method. There I have encountered following update rule: , where ...
FirePapaya's user avatar
2 votes
1 answer
54 views

Calculating derivative for the final layer of a neural network

I'm first learning about backpropagation in neural networks. We're doing stochastic gradient descent. The lecture provides incomplete detail on computing the derivatives for the final layer. We have ...
Ben G's user avatar
  • 153
1 vote
0 answers
38 views

Do common implementations of mini-batch gradient descent violate the i.i.d assumption needed for unbiased estimation?

When we perform mini-batch GD, we estimate the true gradient: $$\nabla L = \frac{1}{N} \sum_i \nabla L_i$$ with: $$\nabla_B L = \frac{1}{B} \sum_{i \in B} \nabla L_i$$ where $B$ is the batch size. ...
ado sar's user avatar
  • 471
0 votes
0 answers
99 views

Convergence in Logistic Regression

Hey I'm taking a deeper dive into logistic regression. Specifically the following loss function with L2 regularization, $$l(w)=\frac{1}{n}\sum_n \log(1+\exp(-y_i \cdot x_i^Tw))+\frac{\lambda}{2}||w||^...
zzz's user avatar
  • 1
1 vote
1 answer
50 views

Trying to reproduce proof of Bandit Gradient Algorithm as SGD

I'm trying to make sense of the "The Bandit Gradient Algorithm as Stochastic Gradient Ascent" proof in Sutton and Barto's intro to RL textbook. I'm stuck on the line $E[(q_*(A_t)-B_t)\frac{\...
fyzr's user avatar
  • 11
5 votes
2 answers
429 views

What is the meaning of "SGD scales the gradient uniformly in all directions"?

I'm really newbie about neural network and optimization. When I read the references, I found this journal Wang et al 2018. The journal stated: One disadvantage of SGD is that it scales the gradient ...
andryan86's user avatar
  • 127
2 votes
1 answer
87 views

What happened to the "other" stochastic gradient descent?

Stochastic gradient descent is a useful approach to improving iteration time by giving up some rate of convergence. For a parameter $w$, learning rate $\eta$, and smooth objective function $Q$ the ...
Galen's user avatar
  • 9,219
1 vote
0 answers
22 views

Could someone help me interpret the data that I have gathered thus far?

I am trying to train a SVM model for my statistical learning course. The problem is a binary image classification problem (wildfire, nowildfire). This is the rigorous amount of testing that I have ...
Ahsan Yousaf's user avatar
2 votes
0 answers
25 views

SGD on finite datasets

In stochastic convex optimization, if $F(w) = E[l(w^Tx,y)]$, when l is a convex, L-Lipschitz loss function, it can be optimized using SGD such that $E[F(\bar{w}_T)] = \frac{1}{T} E[F(w_t)] \leq \min ...
Yakov Dan's user avatar
  • 121
2 votes
1 answer
170 views

Deep Learning - why are we doing the forward pass on the whole dataset when using SGD

When reading the torch.optim documentation of PyTorch (https://pytorch.org/docs/master/optim.html), they do the forward pass on the whole dataset when using SGD but ...
FluidMechanics Potential Flows's user avatar
1 vote
0 answers
593 views

Stochastic Gradient Descent and Unbiased Estimators

In stochastic gradient descent, we utilize the fact that the sampled gradient is an unbiased estimator of the full gradient. That is, with the loss as: $ \nabla \tilde{L} = \nabla \operatorname{...
Victor M's user avatar
  • 319
2 votes
1 answer
294 views

Is this "stochastic gradient method" different from the stochastic gradient descent algorithm?

In a computational statistics book, I found an optimization method to find local minimum of a function. Let's assume that we have a differentiable function $f: \mathbb{R}^2 \longrightarrow \mathbb{R}$....
Neuchâtel's user avatar
1 vote
0 answers
17 views

Why training is smoother when no randomization of batch after each epochs is applied?

I am wondering why training error converges more smoothly when I am not applying randomization after each epochs? After each epoch I am taking the loss obtained in my last minibatch. I see that when ...
yoshcn's user avatar
  • 11
0 votes
1 answer
15 views

Loss is growing between epochs in simple model [duplicate]

I am new to ML and trying to write a very simple model that can predict the future of a function given the past. That is to say, I define a simple cubic function (y = x^3 + x^2 + x) and apply it to a ...
I hate coding's user avatar
1 vote
1 answer
84 views

Why do we need stochastic gradient descent when we can precompute certain results in batch gradient descent?

I have read that batch gradient descent forces this summation at every step of the update, which makes it time consuming. But if we have the following hypothesis function: $$h(x^i) = w_0 + w_1x^i$$ ...
sage76's user avatar
  • 13
2 votes
0 answers
313 views

Which methods can improve model fit through gradient descent?

I am looking to solve the following question: Which methods below can improve model fit (MSE) through gradient descent? Use a high learning rate Use stochastic gradient descent Start with a high ...
Dan's user avatar
  • 21
1 vote
1 answer
171 views

gradient ascent vs gradient descent update rule

I'm trying to understand the differences between the update rule for stochastic gradient ascent and descent. I've read some articles and still don't understand how to calculate the update rule: ...
Cris Tina's user avatar
6 votes
2 answers
867 views

ALS vs SGD in parallelization

So given the standard objective in matrix factorization for collaborative filtering of minimizing: $$ L = \sum_{u,i \in S} (r_{ui}-q_i^Tp_u)^2 + \lambda(\sum_i||q_i^2||+\sum_u||p_u^2||) $$ , where $r_{...
wwyws's user avatar
  • 321
4 votes
1 answer
585 views

Backpropagation in mini-batch stochastic gradient descent with mean squared error loss

Suppose I have an ANN which has one input layer of size $128$, one hidden layer of size $64$ and one output layer of size $10$ for a classification problem. Let's assume we have a training sample of $...
harlem's user avatar
  • 143
1 vote
0 answers
107 views

Which is better: performance stability or high performance in the initial epochs but extremely unstable performance during validation?

I am working on a multiclassification problem using time series data. I am using a hybrid model (such as LSTM, CNN, attention, etc.). I tried two optimizers, ADAM with 0.001 learning rate and SGD with ...
Ahmad's user avatar
  • 161
1 vote
1 answer
87 views

details of stochastic Expectation Maximization (EM) algorithm

I went into a paper, Online EM Algorithm for Latent Data Models (Olivier Cappé & Eric Moulines, 2009). I got confused by the first equation the authors wrote, the Q function: Here the authors ...
C.Wang's user avatar
  • 11
4 votes
1 answer
35 views

What could go wrong if I do gradient descent by class?

For example, when training on CIFAR10, each minibatch typically contains images from all 10 classes (assuming a moderately large batch size such as 64). What could go wrong if I train on a homogeneous ...
nalzok's user avatar
  • 1,797
3 votes
3 answers
970 views

How to find the gradient when a black box I/O function is involved in evaluation of the loss?

I am trying to learn a neural network $NN_\pi$ to minimize the loss function $$ L_{\pi} = || Y_{true} - F(X_{true}, NN_{\pi}(X_{true}) ) ||^2 $$ where $F$ is a black box (I/O) function (we only ...
Mahesh Kumar's user avatar
3 votes
1 answer
606 views

How to handle weighted examples in stochastic gradient descent (with mini-batches)?

Suppose I have $M$ data points $x_i$ and associated weights $w_i > 0$. I want to optimize a function, $$F(\theta) = \frac{1}{M}\sum_i w_i f(x_i;\theta)$$ in the parameters $\theta$. I will assume ...
a06e's user avatar
  • 4,430
1 vote
0 answers
120 views

Regular *negative* spikes in neural network training loss

I am training an ALL-CNN network using the Adam solver. As the figure shows, the testing seems to converge to an acceptable solution, but there are these regular negative spikes during training that ...
Juang Dse's user avatar
2 votes
0 answers
257 views

Mini-Batch Gradient Descent - Large batch size require small learning rate?

Coursera Machine Learning in the Enterprise - Science of Machine Learning and Custom Training says large batch size require smaller LR. However, How should the learning rate change as the batch size ...
mon's user avatar
  • 1,518
1 vote
0 answers
224 views

Derive the gradient matrices w.r.t. W1 and W2 and backprop equation in a Residual Network [closed]

How would I go about deriving gradient matrices w.r.t. W1 and W2 and backpropagation equation in a residual block that is a part of a larger ResNet network with forward propagation expressed as: $$ F(...
Neuro's user avatar
  • 11
1 vote
1 answer
179 views

How does SGD training error decrease in subsequent epochs with non-iid samples when it is recommended that samples in subsequent epochs be iid?

I have been reading the Deep Learning book by Ian Goodfellow and on pg. 277, they mention: It is also crucial that the minibatches be selected randomly. Computing an unbiased estimate of the expected ...
Kunj Mehta's user avatar
2 votes
1 answer
666 views

which training mode is more convenient for small datasets?

I have a regression problem to be solved using one of neural networks models, but I have a small dataset which contains 30 samples. Which training mode is more suitable for such dataset: stochastic or ...
jojo's user avatar
  • 153
2 votes
0 answers
176 views

Bias introduced when using weak shuffling

I have batch learning problem (in this particular case a neural network) where I am training my data in batches, and then repeating for a number of epochs. In Stochastic Gradient Descent, we minimise ...
namiyousef's user avatar
0 votes
1 answer
144 views

About update procedure in data incremental learning

As far as I understood, the idea of data incremental learning consists of keeping the model always up to date. Suppose that we trained a model for user recognition using voice as input. Therefore, the ...
Mas A's user avatar
  • 203
3 votes
1 answer
1k views

Why does ADAM optimization perform well on non-convex functions and bad on convex functions?

I'm currently trying to understand SGD and ADAM optimization, and I understand that ADAM optimization performs well on non-convex loss functions and that SGD performs well on convex loss functions (...
Amaan's user avatar
  • 237
1 vote
0 answers
258 views

Does always gradients in mini-batch SGD have to be unbiased in order to prove convergence?

I am currently reading this paper [1] and [2]. The authors state that: Our analytical results include almost all of the unbiased compression techniques. And also: (i) gradient compression must be ...
Complicated's user avatar
3 votes
1 answer
138 views

Stochastic Gradient Descent Code Check for Least Squares

I have an R-based implementation of the gradient descent and am trying to also get it to work as SGD. The function matches R's lm function when using the entire data set. But, when I sample from the ...
dhc's user avatar
  • 193
1 vote
1 answer
247 views

Performing Linear Regression using Stochastic Gradient Descent, by batches

I am presented with a data set, where I am supposed to perform linear regression on this using SGD. My first instinct would be to train each data point there is until I reach the last one. Only then ...
cgo's user avatar
  • 9,177
4 votes
1 answer
817 views

Why is a 2nd order derivative optimization better for no hidden layer neural networks?

I was reading in this blog. That first order derivative SGD optimization methods are worse for neural networks without hidden layers and 2nd order is better, because that's what regression uses. Why ...
user8714896's user avatar
1 vote
1 answer
528 views

Why does stochastic gradient descent lead us to a minimum at all?

Why do we think that stochastic gradient descent is going to find a minimum at all? I mean on each iteration SGD moves in the direction that reduces only current batch's error (SGD doesn't care about ...
mathgeek's user avatar
  • 551
3 votes
1 answer
713 views

The reason (and intuition) behind why stochastic gradient descent can get stuck on a local minimum

Suppose you want to find $k$ that minimises your cost function $J(k)$. We may want to apply batch gradient descent or stochastic gradient descent. Let's deliberately initialise $k$ with the same ...
mathgeek's user avatar
  • 551
0 votes
0 answers
25 views

Can stochastic gradient descent for Bayesian Inference? [duplicate]

I was looking at the Bayesian MAP estimate formula which is the "argmax(likelihood * prior)". Can this be calculated using stochastic gradient descent? Gradient descent requires knowing the ...
stats_noob's user avatar
1 vote
1 answer
168 views

Is outer product of marginal distribution the "best" mean-field approximation for a joint distribution?

I am certain this has been asked somewhere else, if that's the case, link me and close the thread. I am studying variational inference and mean-field approximation. All the explanations I come across ...
Evan Pu's user avatar
  • 113

1
2 3 4 5